mirror of
https://github.com/arcodange-org/mediabunny.git
synced 2026-09-27 10:53:50 +02:00
Add docs for custom coders
This commit is contained in:
Vanilagy
@@ -41,7 +41,6 @@ export default withMermaid({
|
||||
items: [
|
||||
{ text: 'Packets & samples', link: '/guide/packets-and-samples' },
|
||||
{ text: 'Supported formats & codecs', link: '/guide/supported-formats-and-codecs' },
|
||||
{ text: 'Custom coders', link: '/guide/custom-coders' },
|
||||
],
|
||||
},
|
||||
],
|
||||
|
||||
@@ -78,7 +78,7 @@ There is one media sink which can be used with any `InputTrack`:
|
||||
|
||||
### `EncodedPacketSink`
|
||||
|
||||
This sink can be used to extract raw, [encoded packets](TODO) from media files and is the most elementary media sink. `EncodedPacketSink` is useful if you don't care about the decoded media data (for example, you're only interested in timestamps), or if you want to roll your own decoding logic.
|
||||
This sink can be used to extract raw, [encoded packets](./packets-and-samples#encodedpacket) from media files and is the most elementary media sink. `EncodedPacketSink` is useful if you don't care about the decoded media data (for example, you're only interested in timestamps), or if you want to roll your own decoding logic.
|
||||
|
||||
Start by constructing the sink from any `InputTrack`:
|
||||
```ts
|
||||
@@ -161,7 +161,7 @@ These sinks can only be used with an `InputVideoTrack`.
|
||||
|
||||
### `VideoSampleSink`
|
||||
|
||||
Use this sink to extract decoded [video samples](TODO) (frames) from a video track. The sink will automatically handle the decoding internally.
|
||||
Use this sink to extract decoded [video samples](./packets-and-samples#videosample) (frames) from a video track. The sink will automatically handle the decoding internally.
|
||||
|
||||
::: info
|
||||
All operations of this sink use [presentation order](#decode-vs-presentation-order).
|
||||
@@ -410,7 +410,7 @@ These sinks can only be used with an `InputAudioTrack`.
|
||||
|
||||
### `AudioSampleSink`
|
||||
|
||||
Use this sink to extract decoded [audio samples](TODO) from an audio track. The sink will automatically handle the decoding internally.
|
||||
Use this sink to extract decoded [audio samples](./packets-and-samples#audiosample) from an audio track. The sink will automatically handle the decoding internally.
|
||||
|
||||
Create the sink like so:
|
||||
```ts
|
||||
@@ -427,7 +427,7 @@ The methods for retrieving samples are analogous to those on `VideoSampleSink`.
|
||||
- `samplesAtTimestamps`\
|
||||
Iterates over samples at specific timestamps; see [Sparse iteration](#sparse-iteration).
|
||||
|
||||
These methods yield [`AudioSample`](TODO) instances.
|
||||
These methods yield [`AudioSample`](./packets-and-samples#audiosample) instances.
|
||||
|
||||
For example, let's use this sink to calculate the average loudness of an audio track using [root mean square](https://en.wikipedia.org/wiki/Root_mean_square):
|
||||
```ts
|
||||
|
||||
@@ -124,7 +124,7 @@ Video sources feed data to video tracks on an `Output`. They all extend the abst
|
||||
|
||||
### `VideoSampleSource`
|
||||
|
||||
This source takes [video samples](TODO), encodes them, and passes the encoded data to the output.
|
||||
This source takes [video samples](./packets-and-samples#videosample), encodes them, and passes the encoded data to the output.
|
||||
|
||||
```ts
|
||||
import { VideoSampleSource } from 'mediakit';
|
||||
@@ -185,7 +185,7 @@ If this source is the only MediaStreamTrack source in the `Output`, then the fir
|
||||
|
||||
### `EncodedVideoPacketSource`
|
||||
|
||||
The most barebones of all video sources, this source can be used to directly pipe [encoded packets](TODO) of video data to the output. This source requires that you take care of the encoding process yourself, which enables you to use the WebCodecs API manually or to plug in your own encoding stack. Alternatively, you may retrieve the encoded packets directly by reading them from another media file, allowing you to skip decoding and reencoding video data.
|
||||
The most barebones of all video sources, this source can be used to directly pipe [encoded packets](./packets-and-samples#encodedpacket) of video data to the output. This source requires that you take care of the encoding process yourself, which enables you to use the WebCodecs API manually or to plug in your own encoding stack. Alternatively, you may retrieve the encoded packets directly by reading them from another media file, allowing you to skip decoding and reencoding video data.
|
||||
|
||||
```ts
|
||||
import { EncodedVideoPacketSource } from 'mediakit';
|
||||
@@ -279,7 +279,7 @@ Audio sources feed data to audio tracks on an `Output`. They all extend the abst
|
||||
|
||||
### `AudioSampleSource`
|
||||
|
||||
This source takes [audio samples](TODO), encodes them, and passes the encoded data to the output.
|
||||
This source takes [audio samples](./packets-and-samples#audiosample), encodes them, and passes the encoded data to the output.
|
||||
|
||||
```ts
|
||||
import { AudioSampleSource } from 'mediakit';
|
||||
@@ -334,7 +334,7 @@ If this source is the only MediaStreamTrack source in the `Output`, then the fir
|
||||
|
||||
### `EncodedAudioPacketSource`
|
||||
|
||||
The most barebones of all audio sources, this source can be used to directly pipe [encoded packets](TODO) of audio data to the output. This source requires that you take care of the encoding process yourself, which enables you to use the WebCodecs API manually or to plug in your own encoding stack. Alternatively, you may retrieve the encoded packets directly by reading them from another media file, allowing you to skip decoding and reencoding audio data.
|
||||
The most barebones of all audio sources, this source can be used to directly pipe [encoded packets](./packets-and-samples#encodedpacket) of audio data to the output. This source requires that you take care of the encoding process yourself, which enables you to use the WebCodecs API manually or to plug in your own encoding stack. Alternatively, you may retrieve the encoded packets directly by reading them from another media file, allowing you to skip decoding and reencoding audio data.
|
||||
|
||||
```ts
|
||||
import { EncodedAudioPacketSource } from 'mediakit';
|
||||
|
||||
@@ -40,7 +40,7 @@ Packets and samples in Mediakit correspond directly with concepts of the [WebCod
|
||||
-> `AudioData`
|
||||
|
||||
Since Mediakit makes heavy use of WebCodecs API, its own classes are typically used as wrappers around the WebCodecs classes. However, this wrapping comes with a few benefits:
|
||||
1. **Independence:** This library remains functional even if the WebCodecs API isn't available. Encoders and decoders can be polyfilled using [custom coders](TODO), and the library can run in non-browser contexts such as Node.js.
|
||||
1. **Independence:** This library remains functional even if the WebCodecs API isn't available. Encoders and decoders can be polyfilled using [custom coders](./supported-formats-and-codecs#custom-coders), and the library can run in non-browser contexts such as Node.js.
|
||||
1. **Extensibility:** The wrappers serve as a namespace for additional operations, such as `toAudioBuffer()` on `AudioSample`, or `draw()` on `VideoSample`.
|
||||
1. **Consistency:** While WebCodecs uses integer microsecond timestamps, Mediakit uses floating-point second timestamps everywhere. With these wrappers, all timing information is always in seconds and the user doesn't need to think about unit conversions.
|
||||
|
||||
|
||||
@@ -118,7 +118,7 @@ await track.canDecode(); // => boolean
|
||||
```
|
||||
|
||||
::: info
|
||||
This check also takes [custom decoders](TODO) into account.
|
||||
This check also takes [custom decoders](./supported-formats-and-codecs#custom-decoders) into account.
|
||||
:::
|
||||
|
||||
#### Track timing info
|
||||
|
||||
@@ -16,7 +16,7 @@ Mediakit supports many commonly used media container formats, all of which are s
|
||||
|
||||
Mediakit supports a wide range of video, audio, and subtitle codecs. More specifically, it supports all codecs specified by the WebCodecs API and a few additional PCM codecs out of the box.
|
||||
|
||||
The availability of the codecs provided by the WebCodecs API depends on the browser and cannot be guaranteed by this library. Mediakit provides [special utility functions](#querying-codec-encodability) to check which codecs are able to be encoded. You can also specify [custom coders](./custom-coders) to provide your own encoder/decoder implementation if the browser doesn't support the codec natively.
|
||||
The availability of the codecs provided by the WebCodecs API depends on the browser and thus cannot be guaranteed by this library. Mediakit provides [special utility functions](#querying-codec-encodability) to check which codecs are able to be encoded. You can also specify [custom coders](#custom-coders) to provide your own encoder/decoder implementation if the browser doesn't support the codec natively.
|
||||
|
||||
::: info
|
||||
Mediakit ships with built-in decoders and encoders for all audio PCM codecs, meaning they are always supported.
|
||||
@@ -85,7 +85,7 @@ Not all codecs can be used with all containers. The following table specifies th
|
||||
| `'webvtt'`[^3] | (✓) | | (✓) | (✓) | | | |
|
||||
|
||||
|
||||
[^1]: PCM audio codecs are not supported by MP4. If somebody were to include PCM audio in an MP4 anyway, this library would still be able to read it.
|
||||
[^1]: PCM audio codecs are not supported by MP4. However, if PCM audio is included in an MP4 nonetheless, this library would still be able to read it.
|
||||
[^2]: WebM only supports a small subset of the codecs supported by Matroska. However, this library can technically read all codecs from a WebM that are supported by Matroska.
|
||||
[^3]: WebVTT can only be written, not read.
|
||||
|
||||
@@ -94,7 +94,7 @@ Not all codecs can be used with all containers. The following table specifies th
|
||||
Mediakit provides utility functions that you can use to check if the browser can encode a given codec. Additionally, you
|
||||
can check if a codec is encodable with a specific _configuration_.
|
||||
|
||||
`canEncode` is a general-purpose function that can be called with all codecs and tests encodability using commonly used configurations:
|
||||
`canEncode` tests whether a codec can be encoded using typical settings:
|
||||
```ts
|
||||
import { canEncode } from 'mediakit';
|
||||
|
||||
@@ -136,13 +136,152 @@ These functions also accept optional configuration options:
|
||||
```ts
|
||||
import { getEncodableVideoCodecs } from 'mediakit';
|
||||
|
||||
// Checks only which of AVC, HEVC and VP8 can be encoded at 1280x720 @10Mbps:
|
||||
// Checks only which of AVC, HEVC and VP8 can be encoded at 1920x1080 @10Mbps:
|
||||
getEncodableVideoCodecs(
|
||||
['avc', 'hevc', 'vp8'],
|
||||
{ width: 1920, height: 1080, bitrate: 1e7 },
|
||||
); // => Promise<VideoCodec[]>
|
||||
```
|
||||
|
||||
::: info
|
||||
These checks also take [custom encoders](#custom-encoders) into account.
|
||||
:::
|
||||
|
||||
## Querying codec decodability
|
||||
|
||||
Whether a codec can be decoded depends on the specific codec configuration of an `InputTrack`; you can use its [`canDecode`](./reading-overview#codec-information) method to check if it is decodable.
|
||||
Whether a codec can be decoded depends on the specific codec configuration of an `InputTrack`; you can use its [`canDecode`](./reading-overview#codec-information) method to check.
|
||||
|
||||
## Custom coders
|
||||
|
||||
Mediakit allows you to register your own custom encoders and decoders—useful if you want to polyfill a codec that's not supported in all browsers, or want to use Mediakit outside of an environment with WebCodecs (such as Node.js).
|
||||
|
||||
Encoders and decoders can be registered for [all video and audio codecs](#codecs) supported by the library. It is not possible to add new codecs.
|
||||
|
||||
::: warning
|
||||
Mediakit requires customs encoders and decoders to follow very specific implementation rules. Pay special attention to the parts labeled with "**must**" to ensure compatibility.
|
||||
:::
|
||||
|
||||
### Custom encoders
|
||||
|
||||
To create a custom video or audio encoder, you'll need to create a class which extends `CustomVideoEncoder` or `CustomAudioEncoder`. Then, you **must** register this class using `registerEncoder`:
|
||||
```ts
|
||||
import { CustomAudioEncoder, registerEncoder } from 'mediakit';
|
||||
|
||||
class MyAwesomeMp3Encoder extends CustomAudioEncoder {
|
||||
// ...
|
||||
}
|
||||
registerEncoder(MyAwesomeMp3Encoder);
|
||||
```
|
||||
|
||||
The following properties are available on each encoder instance and are set by the library:
|
||||
```ts
|
||||
class {
|
||||
// For video encoders:
|
||||
codec: VideoCodec;
|
||||
config: VideoEncoderConfig;
|
||||
onPacket: (packet: EncodedPacket, meta?: EncodedVideoChunkMetadata) => unknown;
|
||||
|
||||
// For audio encoders:
|
||||
codec: AudioCodec;
|
||||
config: AudioEncoderConfig;
|
||||
onPacket: (packet: EncodedPacket, meta?: EncodedAudioChunkMetadata) => unknown;
|
||||
}
|
||||
```
|
||||
|
||||
`codec` and `config` specify the concrete codec configuration to use, and `onPacket` is a method that your code **must** call for each encoded packet it creates.
|
||||
|
||||
You **must** implement the following methods in your custom encoder class:
|
||||
```ts
|
||||
class {
|
||||
// For video encoders:
|
||||
static supports(codec: VideoCodec, config: VideoEncoderConfig): boolean;
|
||||
// For audio encoders:
|
||||
static supports(codec: AudioCodec, config: AudioEncoderConfig): boolean;
|
||||
|
||||
init(): Promise<void> | void;
|
||||
encode(sample: VideoSample, options: VideoEncoderEncodeOptions): Promise<void> | void; // For video
|
||||
encode(sample: AudioSample): Promise<void> | void; // For audio
|
||||
flush(): Promise<void> | void;
|
||||
close(): Promise<void> | void;
|
||||
}
|
||||
```
|
||||
- `supports`\
|
||||
This is a *static* method that **must** return `true` if the encoder is able to encode the specified codec, and `false` if not. If it returns `true`, a new instance of your encoder class will be created by the library and will be used for encoding, taking precedence over the default encoders.
|
||||
- `init`\
|
||||
Called by the library after your class is instantiated. Place any initialization logic here.
|
||||
- `encode`\
|
||||
Called for each sample that is to be encoded. The resulting encoded packet **must** then be passed to the `onPacket` method.
|
||||
- `flush`\
|
||||
Called when the encoder is expected to finish the encoding process for all remaining samples that haven't finished encoding yet. This method **must** return/resolve only once all samples passed to `encode` have been fully encoded. It **must** then reset its own internal state to be ready for the next encoding batch.
|
||||
- `close`\
|
||||
Called when the encoder is no longer needed and can release its internal resources.
|
||||
|
||||
::: info
|
||||
All instance methods of the class can return promises. In this case, the library will make sure to *serialize* all method calls such that no two methods ever run concurrently.
|
||||
:::
|
||||
|
||||
::: warning
|
||||
The packets passed to `onPacket` **must** be in [decode order](./media-sinks.md#decode-vs-presentation-order).
|
||||
:::
|
||||
|
||||
### Custom decoders
|
||||
|
||||
To create a custom video or audio decoder, you'll need to create a class which extends `CustomVideoDecoder` or `CustomAudioDecoder`. Then, you **must** register this class using `registerDecoder`:
|
||||
```ts
|
||||
import { CustomAudioDecoder, registerDecoder } from 'mediakit';
|
||||
|
||||
class MyAwesomeMp3Decoder extends CustomAudioDecoder {
|
||||
// ...
|
||||
}
|
||||
registerDecoder(MyAwesomeMp3Decoder);
|
||||
```
|
||||
|
||||
The following properties are available on each decoder instance and are set by the library:
|
||||
```ts
|
||||
class {
|
||||
// For video decoders:
|
||||
codec: VideoCodec;
|
||||
config: VideoDecoderConfig;
|
||||
onSample: (sample: VideoSample) => unknown;
|
||||
|
||||
// For audio decoders:
|
||||
codec: AudioCodec;
|
||||
config: AudioDecoderConfig;
|
||||
onSample: (sample: AudioSample) => unknown;
|
||||
}
|
||||
```
|
||||
|
||||
`codec` and `config` specify the concrete codec configuration to use, and `onSample` is a method that your code **must** call for each video/audio sample it creates.
|
||||
|
||||
You **must** implement the following methods in your custom decoder class:
|
||||
```ts
|
||||
class {
|
||||
// For video decoders:
|
||||
static supports(codec: VideoCodec, config: VideoDecoderConfig): boolean;
|
||||
// For audio decoders:
|
||||
static supports(codec: AudioCodec, config: AudioDecoderConfig): boolean;
|
||||
|
||||
init(): Promise<void> | void;
|
||||
decode(packet: EncodedPacket): Promise<void> | void;
|
||||
flush(): Promise<void> | void;
|
||||
close(): Promise<void> | void;
|
||||
}
|
||||
```
|
||||
- `supports`\
|
||||
This is a *static* method that **must** return `true` if the decoder is able to decode the specified codec, and `false` if not. If it returns `true`, a new instance of your decoder class will be created by the library and will be used for decoding, taking precedence over the default decoders.
|
||||
- `init`\
|
||||
Called by the library after your class is instantiated. Place any initialization logic here.
|
||||
- `decode`\
|
||||
Called for each `EncodedPacket` that is to be decoded. The resulting video or audio sample **must** then be passed to the `onSample` method.
|
||||
- `flush`\
|
||||
Called when the decoder is expected to finish the decoding process for all remaining packets that haven't finished decoding yet. This method **must** return/resolve only once all packets passed to `decode` have been fully decoded. It **must** then reset its own internal state to be ready for the next decoding batch.
|
||||
- `close`\
|
||||
Called when the decoder is no longer needed and can release its internal resources.
|
||||
|
||||
::: info
|
||||
All instance methods of the class can return promises. In this case, the library will make sure to *serialize* all method calls such that no two methods ever run concurrently.
|
||||
:::
|
||||
|
||||
::: warning
|
||||
The samples passed to `onSample` **must** be sorted by increasing timestamp. This especially means if the decoder is decoding a video stream that makes use of [B-frames](./media-sources.md#b-frames), the decoder **must** internally hold on to these frames so it can emit them sorted by presentation timestamp. This strict sorting requirement is reset each time `flush` is called.
|
||||
:::
|
||||
Reference in New Issue
Block a user