diff --git a/dev/convert.html b/dev/convert.html index c49ed91..0482907 100644 --- a/dev/convert.html +++ b/dev/convert.html @@ -21,7 +21,7 @@ chunked: true, chunkSize: 2**20 }); - const outputFormat = new Metamuxer.Mp4OutputFormat({ fastStart: 'fragmented' }); + const outputFormat = new Metamuxer.OggOutputFormat(); const button = document.createElement('button'); button.textContent = 'Cancel'; diff --git a/docs/guide/input-formats.md b/docs/guide/input-formats.md index dcac6b2..e725a5f 100644 --- a/docs/guide/input-formats.md +++ b/docs/guide/input-formats.md @@ -4,6 +4,20 @@ Mediakit supports a wide variety of commonly used container formats for reading - When creating an `Input`, they are used to specify the list of supported container formats. See [Creating a new input](./reading-overview#creating-a-new-input) for more. - Given an existing `Input`, its `getFormat` method returns the *actual* format of the file as an `InputFormat`. +## Input format properties + +Retrieve the full written name of the format like this: +```ts +inputFormat.name; // => 'MP4' +``` + +You can also retrieve the format's base MIME type: +```ts +inputFormat.mimeType; // => 'video/mp4' +``` + +If you want a file's full MIME type, which depends on track codecs, use [`getMimeType`](./reading-overview#reading-file-metadata) on `Input` instead. + ## Input format singletons Since input formats don't require any additional configuration, each input format is directly available as an exported singleton instance: diff --git a/docs/guide/output-formats.md b/docs/guide/output-formats.md index a81c37f..e9f4627 100644 --- a/docs/guide/output-formats.md +++ b/docs/guide/output-formats.md @@ -6,7 +6,7 @@ An _output format_ specifies the container format of the data written by an `Out Many formats also offer *data callbacks*, which are special callbacks that fire for specific data regions in the output file. -### Format properties +### Output format properties All output formats have a common set of properties you can query. @@ -14,6 +14,9 @@ All output formats have a common set of properties you can query. // Get the format's file extension: format.fileExtension; // => '.mp4' +// Get the format's base MIME type: +format.mimeType; // => 'video/mp4' + // Check which codecs can be contained by the format: format.getSupportedCodecs(); // => MediaCodec[] format.getSupportedVideoCodecs(); // => VideoCodec[] diff --git a/docs/guide/supported-formats-and-codecs.md b/docs/guide/supported-formats-and-codecs.md index e7ba13c..eef1d3c 100644 --- a/docs/guide/supported-formats-and-codecs.md +++ b/docs/guide/supported-formats-and-codecs.md @@ -183,7 +183,7 @@ Whether a codec can be decoded depends on the specific codec configuration of an ## Custom coders -Mediakit allows you to register your own custom encoders and decoders—useful if you want to polyfill a codec that's not supported in all browsers, or want to use Mediakit outside of an environment with WebCodecs (such as Node.js). +Mediakit allows you to register your own custom encoders and decoders - useful if you want to polyfill a codec that's not supported in all browsers, or want to use Mediakit outside of an environment with WebCodecs (such as Node.js). Encoders and decoders can be registered for [all video and audio codecs](#codecs) supported by the library. It is not possible to add new codecs. diff --git a/docs/guide/writing-overview.md b/docs/guide/writing-overview.md index 3d43cd0..517baf3 100644 --- a/docs/guide/writing-overview.md +++ b/docs/guide/writing-overview.md @@ -1,6 +1,6 @@ # Writing overview -Mediakit enables you to create media files with very fine levels of control. You can add an arbitrary number of video, audio and subtitle tracks to a media file, and precisely control the timing of media data. This library supports [many output file formats](./output-formats). Using [output targets](#output-targets), you can decide if you want to build up the entire file in memory or stream it out in chunks as it's being created—allowing you to create very large files. +Mediakit enables you to create media files with very fine levels of control. You can add an arbitrary number of video, audio and subtitle tracks to a media file, and precisely control the timing of media data. This library supports [many output file formats](./output-formats). Using [output targets](#output-targets), you can decide if you want to build up the entire file in memory or stream it out in chunks as it's being created - allowing you to create very large files. Mediakit provides many ways to supply media data for output tracks, nicely integrating with the WebCodecs API, but also allowing you to use your own encoding stack if you wish. These [media sources](./media-sources) come in multiple levels of abstraction, enabling easy use for common use cases while still giving you fine-grained control if you need it. @@ -304,8 +304,31 @@ Check the [Output formats](./output-formats) page to see which format configurat --- -If your output format configuration requires packet buffering, make sure to add media data in a somewhat interleaved way to keep memory usage low. For example, if you're creating a 5-minute file, add your data in chunks—10 seconds of video, then 10 seconds of audio, then repeat—instead of first adding all 300 seconds of video followed by all 300 seconds of audio. +If your output format configuration requires packet buffering, make sure to add media data in a somewhat interleaved way to keep memory usage low. For example, if you're creating a 5-minute file, add your data in chunks - 10 seconds of video, then 10 seconds of audio, then repeat - instead of first adding all 300 seconds of video followed by all 300 seconds of audio. ::: info If this kind of chunking isn't possible for your use case, try adding the media with the overall smaller data footprint first: First add the 300 seconds of audio, then add the 300 seconds of video. ::: + +## Output MIME type + +Sometimes you may want to retrieve the MIME type of the file created by an `Output`. For example, when working with Media Source Extensions, [`addSourceBuffer`](https://developer.mozilla.org/en-US/docs/Web/API/MediaSource/addSourceBuffer) requires the file's full MIME type, including codec strings. + +For this, use the following method: +```ts +output.getMimeType(); // => Promise +``` + +This may resolve to a string like this: +``` +video/mp4; codecs="avc1.42c032, mp4a.40.2" +``` + +::: warning +The promise returned by `getMimeType` only resolves once the precise codec strings for all tracks of the `Output` are known - meaning it potentially needs to wait for all encoders to be fully initialized. Therefore, make sure not to get yourself into a deadlock: Awaiting this method before adding media data to tracks will result in the promise never resolving. +::: + +If you don't care about specific track codecs, you can instead use the simpler [`mimeType`](./output-formats#output-format-properties) property on the `Output`'s format: +```ts +output.format.mimeType; // => string +``` \ No newline at end of file diff --git a/src/codec.ts b/src/codec.ts index 564d797..a275500 100644 --- a/src/codec.ts +++ b/src/codec.ts @@ -240,7 +240,7 @@ export const buildVideoCodecString = (codec: VideoCodec, width: number, height: const bitDepth = '08'; // 8-bit - return `vp09.${profile}.${levelInfo.level.toString().padStart(2, '0')}.${bitDepth}${VP9_DEFAULT_SUFFIX}`; + return `vp09.${profile}.${levelInfo.level.toString().padStart(2, '0')}.${bitDepth}`; } else if (codec === 'av1') { const profile = 0; // Main Profile, single digit @@ -252,7 +252,7 @@ export const buildVideoCodecString = (codec: VideoCodec, width: number, height: const bitDepth = '08'; // 8-bit - return `av01.${profile}.${level}${levelInfo.tier}.${bitDepth}${AV1_DEFAULT_SUFFIX}`; + return `av01.${profile}.${level}${levelInfo.tier}.${bitDepth}`; } // eslint-disable-next-line @typescript-eslint/restrict-template-expressions @@ -426,9 +426,8 @@ export const extractVideoCodecString = (trackInfo: { let string = `vp09.${profile}.${level}.${bitDepth}.${chromaSubsampling}`; string += `.${colourPrimaries}.${transferCharacteristics}.${matrixCoefficients}.${videoFullRangeFlag}`; - const defaultSuffix = '.01.01.01.01.00'; - if (string.endsWith(defaultSuffix)) { - string = string.slice(0, -defaultSuffix.length); + if (string.endsWith(VP9_DEFAULT_SUFFIX)) { + string = string.slice(0, -VP9_DEFAULT_SUFFIX.length); } return string; @@ -476,9 +475,8 @@ export const extractVideoCodecString = (trackInfo: { string += `.${matrixCoefficients.toString().padStart(2, '0')}`; string += `.${videoFullRangeFlag}`; - const defaultSuffix = '.0.110.01.01.01.0'; - if (string.endsWith(defaultSuffix)) { - string = string.slice(0, -defaultSuffix.length); + if (string.endsWith(AV1_DEFAULT_SUFFIX)) { + string = string.slice(0, -AV1_DEFAULT_SUFFIX.length); } return string; diff --git a/src/conversion.ts b/src/conversion.ts index addaa1c..42606fd 100644 --- a/src/conversion.ts +++ b/src/conversion.ts @@ -111,13 +111,14 @@ const FALLBACK_SAMPLE_RATE = 48000; * @public */ export class Conversion { + /** The input file. */ + input: Input; + /** The output file. */ + output: Output; + /** @internal */ _options: ConversionOptions; /** @internal */ - _input: Input; - /** @internal */ - _output: Output; - /** @internal */ _startTimestamp: number; /** @internal */ _endTimestamp: number; @@ -299,8 +300,8 @@ export class Conversion { } this._options = options; - this._input = options.input; - this._output = options.output; + this.input = options.input; + this.output = options.output; this._startTimestamp = options.trim?.start ?? 0; this._endTimestamp = options.trim?.end ?? Infinity; @@ -312,8 +313,8 @@ export class Conversion { /** @internal */ async _init() { - const inputTracks = await this._input.getTracks(); - const outputTrackCounts = this._output.format.getSupportedTrackCounts(); + const inputTracks = await this.input.getTracks(); + const outputTrackCounts = this.output.format.getSupportedTrackCounts(); for (const track of inputTracks) { if (track.isVideoTrack() && this._options.video?.discard) { @@ -373,13 +374,13 @@ export class Conversion { if (this.onProgress) { this._computeProgress = true; this._totalDuration = Math.min( - await this._input.computeDuration() - this._startTimestamp, + await this.input.computeDuration() - this._startTimestamp, this._endTimestamp - this._startTimestamp, ); this.onProgress?.(0); } - await this._output.start(); + await this.output.start(); this._start(); try { @@ -397,7 +398,7 @@ export class Conversion { await new Promise(() => {}); // Never resolve } - await this._output.finalize(); + await this.output.finalize(); if (this._computeProgress) { this.onProgress?.(1); @@ -406,7 +407,7 @@ export class Conversion { /** Cancels the conversion process. Does nothing if the conversion is already complete. */ async cancel() { - if (this._output.state === 'finalizing' || this._output.state === 'finalized') { + if (this.output.state === 'finalizing' || this.output.state === 'finalized') { return; } @@ -416,7 +417,7 @@ export class Conversion { } this._canceled = true; - await this._output.cancel(); + await this.output.cancel(); } /** @internal */ @@ -433,7 +434,7 @@ export class Conversion { let videoSource: VideoSource; const totalRotation = normalizeRotation(track.rotation + (this._options.video?.rotate ?? 0)); - const outputSupportsRotation = this._output.format.supportsVideoRotationMetadata; + const outputSupportsRotation = this.output.format.supportsVideoRotationMetadata; const [originalWidth, originalHeight] = totalRotation % 180 === 0 ? [track.codedWidth, track.codedHeight] @@ -463,7 +464,7 @@ export class Conversion { || height !== originalHeight || (totalRotation !== 0 && !outputSupportsRotation); - let videoCodecs = this._output.format.getSupportedVideoCodecs(); + let videoCodecs = this.output.format.getSupportedVideoCodecs(); if ( !needsReencode && !this._options.video?.bitrate @@ -599,7 +600,7 @@ export class Conversion { } } - this._output.addVideoTrack(videoSource, { + this.output.addVideoTrack(videoSource, { languageCode: track.languageCode, rotation: needsRerender ? 0 : totalRotation, // Rerendering will bake the rotation into the output }); @@ -634,7 +635,7 @@ export class Conversion { || this._startTimestamp > 0 || firstTimestamp < 0; - let audioCodecs = this._output.format.getSupportedAudioCodecs(); + let audioCodecs = this.output.format.getSupportedAudioCodecs(); if ( !this._options.audio?.forceReencode && !this._options.audio?.bitrate @@ -768,7 +769,7 @@ export class Conversion { } } - this._output.addAudioTrack(audioSource, { + this.output.addAudioTrack(audioSource, { languageCode: track.languageCode, }); this._addedCounts.audio++; diff --git a/src/input-format.ts b/src/input-format.ts index 6e61ad6..c159794 100644 --- a/src/input-format.ts +++ b/src/input-format.ts @@ -23,9 +23,9 @@ export abstract class InputFormat { abstract _createDemuxer(input: Input): Demuxer; /** Returns the name of the input format. */ - abstract getName(): string; + abstract get name(): string; /** Returns the typical base MIME type of the input format. */ - abstract getMimeType(): string; + abstract get mimeType(): string; } /** @@ -68,11 +68,11 @@ export class Mp4InputFormat extends IsobmffInputFormat { return !!majorBrand && majorBrand !== 'qt '; } - getName() { + get name() { return 'MP4'; } - getMimeType() { + get mimeType() { return 'video/mp4'; } } @@ -88,11 +88,11 @@ export class QuickTimeInputFormat extends IsobmffInputFormat { return majorBrand === 'qt '; } - getName() { + get name() { return 'QuickTime File Format'; } - getMimeType() { + get mimeType() { return 'video/quicktime'; } } @@ -174,11 +174,11 @@ export class MatroskaInputFormat extends InputFormat { return new MatroskaDemuxer(input); } - getName() { + get name() { return 'Matroska'; } - getMimeType() { + get mimeType() { return 'video/x-matroska'; } } @@ -193,11 +193,11 @@ export class WebMInputFormat extends MatroskaInputFormat { return this.isSupportedEBMLOfDocType(input, 'webm'); } - override getName() { + override get name() { return 'WebM'; } - override getMimeType() { + override get mimeType() { return 'video/webm'; } } @@ -258,11 +258,11 @@ export class Mp3InputFormat extends InputFormat { return new Mp3Demuxer(input); } - getName() { + get name() { return 'MP3'; } - getMimeType() { + get mimeType() { return 'audio/mpeg'; } } @@ -295,11 +295,11 @@ export class WaveInputFormat extends InputFormat { return new WaveDemuxer(input); } - getName() { + get name() { return 'WAVE'; } - getMimeType() { + get mimeType() { return 'audio/wav'; } } @@ -325,11 +325,11 @@ export class OggInputFormat extends InputFormat { return new OggDemuxer(input); } - getName() { + get name() { return 'Ogg'; } - getMimeType() { + get mimeType() { return 'application/ogg'; } } diff --git a/src/isobmff/isobmff-demuxer.ts b/src/isobmff/isobmff-demuxer.ts index 83256a0..8effbad 100644 --- a/src/isobmff/isobmff-demuxer.ts +++ b/src/isobmff/isobmff-demuxer.ts @@ -49,6 +49,7 @@ import { } from '../misc'; import { EncodedPacket, PLACEHOLDER_DATA } from '../packet'; import { Reader } from '../reader'; +import { buildIsobmffMimeType } from './isobmff-misc'; import { IsobmffReader, MAX_BOX_HEADER_SIZE } from './isobmff-reader'; type InternalTrack = { @@ -223,22 +224,14 @@ export class IsobmffDemuxer extends Demuxer { override async getMimeType() { await this.readMetadata(); - const base = this.tracks.some(x => x.info?.type === 'video') - ? 'video/' - : this.tracks.some(x => x.info?.type === 'audio') - ? 'audio/' - : 'application/'; + const codecStrings = await Promise.all(this.tracks.map(x => x.inputTrack!.getCodecParameterString())); - let string = base + (this.isQuickTime ? 'quicktime' : 'mp4'); - - if (this.tracks.length > 0) { - const codecMimeTypes = await Promise.all(this.tracks.map(x => x.inputTrack!.getCodecParameterString())); - const uniqueCodecMimeTypes = [...new Set(codecMimeTypes.filter(Boolean))]; - - string += `; codecs="${uniqueCodecMimeTypes.join(', ')}"`; - } - - return string; + return buildIsobmffMimeType({ + isQuicktime: this.isQuickTime, + hasVideo: this.tracks.some(x => x.info?.type === 'video'), + hasAudio: this.tracks.some(x => x.info?.type === 'audio'), + codecStrings: codecStrings.filter(Boolean) as string[], + }); } readMetadata() { diff --git a/src/isobmff/isobmff-misc.ts b/src/isobmff/isobmff-misc.ts new file mode 100644 index 0000000..804932d --- /dev/null +++ b/src/isobmff/isobmff-misc.ts @@ -0,0 +1,21 @@ +export const buildIsobmffMimeType = (info: { + isQuicktime: boolean; + hasVideo: boolean; + hasAudio: boolean; + codecStrings: string[]; +}) => { + const base = info.hasVideo + ? 'video/' + : info.hasAudio + ? 'audio/' + : 'application/'; + + let string = base + (info.isQuicktime ? 'quicktime' : 'mp4'); + + if (info.codecStrings.length > 0) { + const uniqueCodecMimeTypes = [...new Set(info.codecStrings)]; + string += `; codecs="${uniqueCodecMimeTypes.join(', ')}"`; + } + + return string; +}; diff --git a/src/isobmff/isobmff-muxer.ts b/src/isobmff/isobmff-muxer.ts index 234f349..8294e54 100644 --- a/src/isobmff/isobmff-muxer.ts +++ b/src/isobmff/isobmff-muxer.ts @@ -2,13 +2,14 @@ import { Box, ftyp, IsobmffBoxWriter, mdat, mfra, moof, moov, vtta, vttc, vtte } import { Muxer } from '../muxer'; import { Output, OutputAudioTrack, OutputSubtitleTrack, OutputTrack, OutputVideoTrack } from '../output'; import { BufferTargetWriter, Writer } from '../writer'; -import { assert, computeRationalApproximation, last } from '../misc'; +import { assert, computeRationalApproximation, last, promiseWithResolvers } from '../misc'; import { IsobmffOutputFormatOptions, IsobmffOutputFormat, MovOutputFormat } from '../output-format'; import { inlineTimestampRegex, SubtitleConfig, SubtitleCue, SubtitleMetadata } from '../subtitles'; import { parsePcmCodec, PCM_AUDIO_CODECS, PcmAudioCodec, + SubtitleCodec, validateAudioChunkMetadata, validateSubtitleMetadata, validateVideoChunkMetadata, @@ -22,6 +23,7 @@ import { serializeHevcDecoderConfigurationRecord, transformAnnexBToLengthPrefixed, } from '../codec-data'; +import { buildIsobmffMimeType } from './isobmff-misc'; export const GLOBAL_TIMESCALE = 1000; const TIMESTAMP_OFFSET = 2_082_844_800; // Seconds between Jan 1 1904 and Jan 1 1970 @@ -125,6 +127,7 @@ export class IsobmffMuxer extends Muxer { private mdat: Box | null = null; private trackDatas: IsobmffTrackData[] = []; + private allTracksKnown = promiseWithResolvers(); private creationTime = Math.floor(Date.now() / 1000) + TIMESTAMP_OFFSET; private finalizedChunks: Chunk[] = []; @@ -197,6 +200,40 @@ export class IsobmffMuxer extends Muxer { release(); } + private allTracksAreKnown() { + for (const track of this.output._tracks) { + if (!track.source._closed && !this.trackDatas.some(x => x.track === track)) { + return false; // We haven't seen a sample from this open track yet + } + } + + return true; + } + + async getMimeType() { + await this.allTracksKnown.promise; + + const codecStrings = this.trackDatas.map((trackData) => { + if (trackData.type === 'video') { + return trackData.info.decoderConfig.codec; + } else if (trackData.type === 'audio') { + return trackData.info.decoderConfig.codec; + } else { + const map: Record = { + webvtt: 'wvtt', + }; + return map[trackData.track.source._codec]; + } + }); + + return buildIsobmffMimeType({ + isQuicktime: this.isMov, + hasVideo: this.trackDatas.some(x => x.type === 'video'), + hasAudio: this.trackDatas.some(x => x.type === 'audio'), + codecStrings, + }); + } + private getVideoTrackData(track: OutputVideoTrack, packet: EncodedPacket, meta?: EncodedVideoChunkMetadata) { const existingTrackData = this.trackDatas.find(x => x.track === track); if (existingTrackData) { @@ -278,6 +315,10 @@ export class IsobmffMuxer extends Muxer { this.trackDatas.push(newTrackData); this.trackDatas.sort((a, b) => a.track.id - b.track.id); + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + return newTrackData; } @@ -319,6 +360,10 @@ export class IsobmffMuxer extends Muxer { this.trackDatas.push(newTrackData); this.trackDatas.sort((a, b) => a.track.id - b.track.id); + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + return newTrackData; } @@ -360,6 +405,10 @@ export class IsobmffMuxer extends Muxer { this.trackDatas.push(newTrackData); this.trackDatas.sort((a, b) => a.track.id - b.track.id); + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + return newTrackData; } @@ -869,10 +918,8 @@ export class IsobmffMuxer extends Muxer { assert(this.isFragmented); if (!isFinalCall) { - for (const track of this.output._tracks) { - if (!track.source._closed && !this.trackDatas.some(x => x.track === track)) { - return; // We haven't seen a sample from this open track yet - } + if (!this.allTracksAreKnown()) { + return; // We can't interleave yet as we don't yet know how many tracks we'll truly have } } @@ -1006,6 +1053,10 @@ export class IsobmffMuxer extends Muxer { } } + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + if (this.isFragmented) { // Since a track is now closed, we may be able to write out chunks that were previously waiting await this.interleaveSamples(); @@ -1018,6 +1069,8 @@ export class IsobmffMuxer extends Muxer { async finalize() { const release = await this.mutex.acquire(); + this.allTracksKnown.resolve(); + for (const trackData of this.trackDatas) { if (trackData.type === 'subtitle' && trackData.track.source._codec === 'webvtt') { await this.processWebVTTCues(trackData, Infinity); diff --git a/src/matroska/matroska-demuxer.ts b/src/matroska/matroska-demuxer.ts index 7b5000f..d60d373 100644 --- a/src/matroska/matroska-demuxer.ts +++ b/src/matroska/matroska-demuxer.ts @@ -50,6 +50,7 @@ import { MAX_HEADER_SIZE, MIN_HEADER_SIZE, } from './ebml'; +import { buildMatroskaMimeType } from './matroska-misc'; type Segment = { seekHeadSeen: boolean; @@ -194,23 +195,15 @@ export class MatroskaDemuxer extends Demuxer { override async getMimeType() { await this.readMetadata(); - const base = this.segments.some(segment => segment.tracks.some(x => x.info?.type === 'video')) - ? 'video/' - : this.segments.some(segment => segment.tracks.some(x => x.info?.type === 'audio')) - ? 'audio/' - : 'application/'; - - let string = base + (this.isWebM ? 'webm' : 'x-matroska'); - const tracks = await this.getTracks(); - if (tracks.length > 0) { - const codecMimeTypes = await Promise.all(tracks.map(x => x.getCodecParameterString())); - const uniqueCodecMimeTypes = [...new Set(codecMimeTypes.filter(Boolean))]; + const codecStrings = await Promise.all(tracks.map(x => x.getCodecParameterString())); - string += `; codecs="${uniqueCodecMimeTypes.join(', ')}"`; - } - - return string; + return buildMatroskaMimeType({ + isWebM: this.isWebM, + hasVideo: this.segments.some(segment => segment.tracks.some(x => x.info?.type === 'video')), + hasAudio: this.segments.some(segment => segment.tracks.some(x => x.info?.type === 'audio')), + codecStrings: codecStrings.filter(Boolean) as string[], + }); } readMetadata() { diff --git a/src/matroska/matroska-misc.ts b/src/matroska/matroska-misc.ts new file mode 100644 index 0000000..6a8dc8d --- /dev/null +++ b/src/matroska/matroska-misc.ts @@ -0,0 +1,21 @@ +export const buildMatroskaMimeType = (info: { + isWebM: boolean; + hasVideo: boolean; + hasAudio: boolean; + codecStrings: string[]; +}) => { + const base = info.hasVideo + ? 'video/' + : info.hasAudio + ? 'audio/' + : 'application/'; + + let string = base + (info.isWebM ? 'webm' : 'x-matroska'); + + if (info.codecStrings.length > 0) { + const uniqueCodecMimeTypes = [...new Set(info.codecStrings.filter(Boolean))]; + string += `; codecs="${uniqueCodecMimeTypes.join(', ')}"`; + } + + return string; +}; diff --git a/src/matroska/matroska-muxer.ts b/src/matroska/matroska-muxer.ts index f56a750..f300e61 100644 --- a/src/matroska/matroska-muxer.ts +++ b/src/matroska/matroska-muxer.ts @@ -7,6 +7,7 @@ import { assert, colorSpaceIsComplete, normalizeRotation, + promiseWithResolvers, roundToMultiple, textEncoder, toUint8Array, @@ -22,6 +23,7 @@ import { EBMLSignedInt, EBMLWriter, } from './ebml'; +import { buildMatroskaMimeType } from './matroska-misc'; import { MkvOutputFormat, WebMOutputFormat } from '../output-format'; import { Output, OutputAudioTrack, OutputSubtitleTrack, OutputTrack, OutputVideoTrack } from '../output'; import { @@ -36,6 +38,7 @@ import { OPUS_INTERNAL_SAMPLE_RATE, PCM_AUDIO_CODECS, PcmAudioCodec, + SubtitleCodec, generateAv1CodecConfigurationFromCodecString, generateVp9CodecConfigurationFromCodecString, parsePcmCodec, @@ -121,6 +124,7 @@ export class MatroskaMuxer extends Muxer { private format: WebMOutputFormat | MkvOutputFormat; private trackDatas: MatroskaTrackData[] = []; + private allTracksKnown = promiseWithResolvers(); private segment: EBMLElement | null = null; private segmentInfo: EBMLElement | null = null; @@ -396,6 +400,40 @@ export class MatroskaMuxer extends Muxer { return this.ebmlWriter.dataOffsets.get(this.segment)!; } + private allTracksAreKnown() { + for (const track of this.output._tracks) { + if (!track.source._closed && !this.trackDatas.some(x => x.track === track)) { + return false; // We haven't seen a sample from this open track yet + } + } + + return true; + } + + async getMimeType() { + await this.allTracksKnown.promise; + + const codecStrings = this.trackDatas.map((trackData) => { + if (trackData.type === 'video') { + return trackData.info.decoderConfig.codec; + } else if (trackData.type === 'audio') { + return trackData.info.decoderConfig.codec; + } else { + const map: Record = { + webvtt: 'wvtt', + }; + return map[trackData.track.source._codec]; + } + }); + + return buildMatroskaMimeType({ + isWebM: this.format instanceof WebMOutputFormat, + hasVideo: this.trackDatas.some(x => x.type === 'video'), + hasAudio: this.trackDatas.some(x => x.type === 'audio'), + codecStrings, + }); + } + private getVideoTrackData(track: OutputVideoTrack, meta?: EncodedVideoChunkMetadata) { const existingTrackData = this.trackDatas.find(x => x.track === track); if (existingTrackData) { @@ -445,6 +483,10 @@ export class MatroskaMuxer extends Muxer { this.trackDatas.push(newTrackData); this.trackDatas.sort((a, b) => a.track.id - b.track.id); + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + return newTrackData; } @@ -474,6 +516,10 @@ export class MatroskaMuxer extends Muxer { this.trackDatas.push(newTrackData); this.trackDatas.sort((a, b) => a.track.id - b.track.id); + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + return newTrackData; } @@ -501,6 +547,10 @@ export class MatroskaMuxer extends Muxer { this.trackDatas.push(newTrackData); this.trackDatas.sort((a, b) => a.track.id - b.track.id); + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + return newTrackData; } @@ -587,10 +637,8 @@ export class MatroskaMuxer extends Muxer { private async interleaveChunks(isFinalCall = false) { if (!isFinalCall) { - for (const track of this.output._tracks) { - if (!track.source._closed && !this.trackDatas.some(x => x.track === track)) { - return; // We haven't seen a sample from this open track yet - } + if (!this.allTracksAreKnown()) { + return; // We can't interleave yet as we don't yet know how many tracks we'll truly have } } @@ -880,6 +928,10 @@ export class MatroskaMuxer extends Muxer { override async onTrackClose() { const release = await this.mutex.acquire(); + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + // Since a track is now closed, we may be able to write out chunks that were previously waiting await this.interleaveChunks(); @@ -890,6 +942,8 @@ export class MatroskaMuxer extends Muxer { async finalize() { const release = await this.mutex.acquire(); + this.allTracksKnown.resolve(); + if (!this.segment) { this.createTracks(); this.createSegment(); diff --git a/src/mp3/mp3-muxer.ts b/src/mp3/mp3-muxer.ts index 9f465e6..80bdcd7 100644 --- a/src/mp3/mp3-muxer.ts +++ b/src/mp3/mp3-muxer.ts @@ -27,6 +27,10 @@ export class Mp3Muxer extends Muxer { // Nothing needed here } + async getMimeType() { + return 'audio/mpeg'; + } + async addEncodedVideoPacket() { throw new Error('MP3 does not support video.'); } diff --git a/src/muxer.ts b/src/muxer.ts index 77b88ba..16948fd 100644 --- a/src/muxer.ts +++ b/src/muxer.ts @@ -19,6 +19,7 @@ export abstract class Muxer { } abstract start(): Promise; + abstract getMimeType(): Promise; abstract addEncodedVideoPacket( track: OutputVideoTrack, packet: EncodedPacket, diff --git a/src/ogg/ogg-demuxer.ts b/src/ogg/ogg-demuxer.ts index 7740c24..8fbe31e 100644 --- a/src/ogg/ogg-demuxer.ts +++ b/src/ogg/ogg-demuxer.ts @@ -7,7 +7,7 @@ import { PacketRetrievalOptions } from '../media-sink'; import { assert, AsyncMutex, findLast, roundToPrecision, toDataView, UNDETERMINED_LANGUAGE } from '../misc'; import { EncodedPacket, PLACEHOLDER_DATA } from '../packet'; import { Reader } from '../reader'; -import { computeOggPageCrc, extractSampleMetadata, OggCodecInfo } from './ogg-misc'; +import { buildOggMimeType, computeOggPageCrc, extractSampleMetadata, OggCodecInfo } from './ogg-misc'; import { MAX_PAGE_HEADER_SIZE, MAX_PAGE_SIZE, MIN_PAGE_HEADER_SIZE, OggReader, Page } from './ogg-reader'; type LogicalBitstream = { @@ -366,16 +366,11 @@ export class OggDemuxer extends Demuxer { async getMimeType() { await this.readMetadata(); - let string = 'audio/ogg'; + const codecStrings = await Promise.all(this.tracks.map(x => x.getCodecParameterString())); - if (this.tracks.length > 0) { - const codecMimeTypes = await Promise.all(this.tracks.map(x => x.getCodecParameterString())); - const uniqueCodecMimeTypes = [...new Set(codecMimeTypes.filter(Boolean))]; - - string += `; codecs="${uniqueCodecMimeTypes.join(', ')}"`; - } - - return string; + return buildOggMimeType({ + codecStrings: codecStrings.filter(Boolean) as string[], + }); } async getTracks() { diff --git a/src/ogg/ogg-misc.ts b/src/ogg/ogg-misc.ts index 83745c5..2785e4c 100644 --- a/src/ogg/ogg-misc.ts +++ b/src/ogg/ogg-misc.ts @@ -93,3 +93,16 @@ export const extractSampleMetadata = ( vorbisBlockSize: currentBlocksize, }; }; + +export const buildOggMimeType = (info: { + codecStrings: string[]; +}) => { + let string = 'audio/ogg'; + + if (info.codecStrings) { + const uniqueCodecMimeTypes = [...new Set(info.codecStrings)]; + string += `; codecs="${uniqueCodecMimeTypes.join(', ')}"`; + } + + return string; +}; diff --git a/src/ogg/ogg-muxer.ts b/src/ogg/ogg-muxer.ts index 0f4c0ca..3d437d1 100644 --- a/src/ogg/ogg-muxer.ts +++ b/src/ogg/ogg-muxer.ts @@ -1,12 +1,13 @@ import { OPUS_INTERNAL_SAMPLE_RATE, validateAudioChunkMetadata } from '../codec'; import { parseModesFromVorbisSetupPacket, parseOpusIdentificationHeader } from '../codec-data'; -import { assert, setInt64, toDataView, toUint8Array } from '../misc'; +import { assert, promiseWithResolvers, setInt64, toDataView, toUint8Array } from '../misc'; import { Muxer } from '../muxer'; import { Output, OutputAudioTrack } from '../output'; import { OggOutputFormat } from '../output-format'; import { EncodedPacket } from '../packet'; import { Writer } from '../writer'; import { + buildOggMimeType, computeOggPageCrc, extractSampleMetadata, OggCodecInfo, @@ -46,6 +47,7 @@ export class OggMuxer extends Muxer { private trackDatas: OggTrackData[] = []; private bosPagesWritten = false; + private allTracksKnown = promiseWithResolvers(); private pageBytes = new Uint8Array(MAX_PAGE_SIZE); private pageView = new DataView(this.pageBytes.buffer); @@ -63,6 +65,14 @@ export class OggMuxer extends Muxer { // Nothin' } + async getMimeType() { + await this.allTracksKnown.promise; + + return buildOggMimeType({ + codecStrings: this.trackDatas.map(x => x.codecInfo.codec!), + }); + } + addEncodedVideoPacket(): never { throw new Error('Video tracks are not supported.'); } @@ -112,6 +122,11 @@ export class OggMuxer extends Muxer { this.queueHeaderPackets(newTrackData, meta); this.trackDatas.push(newTrackData); + + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + return newTrackData; } @@ -250,12 +265,20 @@ export class OggMuxer extends Muxer { throw new Error('Subtitle tracks are not supported.'); } + allTracksAreKnown() { + for (const track of this.output._tracks) { + if (!track.source._closed && !this.trackDatas.some(x => x.track === track)) { + return false; // We haven't seen a sample from this open track yet + } + } + + return true; + } + async interleavePages(isFinalCall = false) { if (!this.bosPagesWritten) { - for (const track of this.output._tracks) { - if (!track.source._closed && !this.trackDatas.some(x => x.track === track)) { - return; // We haven't seen a sample from this open track yet - } + if (!this.allTracksAreKnown()) { + return; // We can't interleave yet as we don't yet know how many tracks we'll truly have } // Write the header page for all bitstreams @@ -421,6 +444,10 @@ export class OggMuxer extends Muxer { override async onTrackClose() { const release = await this.mutex.acquire(); + if (this.allTracksAreKnown()) { + this.allTracksKnown.resolve(); + } + // Since a track is now closed, we may be able to write out chunks that were previously waiting await this.interleavePages(); @@ -430,6 +457,8 @@ export class OggMuxer extends Muxer { async finalize() { const release = await this.mutex.acquire(); + this.allTracksKnown.resolve(); + await this.interleavePages(true); for (const trackData of this.trackDatas) { diff --git a/src/output.ts b/src/output.ts index 7538b56..4bd0a11 100644 --- a/src/output.ts +++ b/src/output.ts @@ -344,6 +344,15 @@ export class Output< })(); } + /** + * Resolves with the full MIME type of the output file, including track codecs. + * + * The returned promise will resolve only once the precise codec strings of all tracks are known. + */ + getMimeType() { + return this._muxer.getMimeType(); + } + /** * Cancels the creation of the output file, releasing internal resources like encoders and preventing further * samples from being added. diff --git a/src/wave/wave-muxer.ts b/src/wave/wave-muxer.ts index 99641d9..7147e56 100644 --- a/src/wave/wave-muxer.ts +++ b/src/wave/wave-muxer.ts @@ -27,6 +27,10 @@ export class WaveMuxer extends Muxer { // Nothing needed here - we'll write the header with the first sample } + async getMimeType() { + return 'audio/wav'; + } + async addEncodedVideoPacket() { throw new Error('WAVE does not support video.'); }