mirror of
https://github.com/arcodange-org/mediabunny.git
synced 2026-09-27 02:43:48 +02:00
Add toAvFrame, NodeAv* -> Av*, add server README, fix track synchronizer in conversion, increase default cache size for ReadableStreamSource
This commit is contained in:
Vanilagy
+276
-1
@@ -1 +1,276 @@
|
||||
todo
|
||||
# @mediabunny/server
|
||||
|
||||
[](https://www.npmjs.com/package/@mediabunny/server)
|
||||
[](https://bundlephobia.com/package/@mediabunny/server)
|
||||
[](https://www.npmjs.com/package/@mediabunny/server)
|
||||
[](https://discord.gg/hmpkyYuS4U)
|
||||
|
||||
<div align="center">
|
||||
<img src="./logo.svg" width="180" height="180">
|
||||
</div>
|
||||
|
||||
By default, Mediabunny requires a browser environment for full access to decoders, encoders, and video processing features. `@mediabunny/server` uses [NodeAV](https://github.com/seydx/node-av) to polyfill this functionality for server-side environments such as Node, Bun, or Deno, enabling the usage of all Mediabunny features on the server. The result is a server-side media processing API that integrates naturally with TypeScript as opposed to the awkwardness and inefficiencies of calling out to the FFmpeg CLI.
|
||||
|
||||
Features added by this package include:
|
||||
- Video decoders and encoders for AVC (H.264), HEVC (H.265), VP8, VP9, and AV1. Supports both length-prefixed and Annex B AVC/HEVC as well as transparent video via VP9.
|
||||
- Audio decoders and encoders for AAC, MP3, Vorbis, Opus, FLAC, AC-3 and E-AC-3. Supports AAC in both AAC and ADTS formats.
|
||||
- Video frame transformation support (resize, rotate, crop)
|
||||
- Automatic hardware acceleration on all platforms (macOS, Linux, Windows)
|
||||
- Built-in multithreading
|
||||
- Zero-copy decode and encode paths
|
||||
|
||||
> This package, like the rest of Mediabunny, is enabled by its [sponsors](https://mediabunny.dev/#sponsors) and their donations. If you've derived value from this package, please consider [leaving a donation](https://github.com/sponsors/Vanilagy)! 💘
|
||||
|
||||
> This package was made possible in large part due to seydx's amazing work on [NodeAV](https://github.com/seydx/node-av). The library is truly modern, a joy to work with, and incredibly powerful. Give them a star!
|
||||
|
||||
## Installation
|
||||
|
||||
This library peer-depends on Mediabunny. Install both using npm:
|
||||
```bash
|
||||
npm install mediabunny @mediabunny/server
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```ts
|
||||
import { registerMediabunnyServer } from '@mediabunny/server';
|
||||
registerMediabunnyServer();
|
||||
```
|
||||
|
||||
That's it - you now have access to the full Mediabunny feature set on the server.
|
||||
|
||||
## Upload media compression example
|
||||
|
||||
Here, we set up a simple media compression server in Node.js. The client's request body is streamed to Mediabunny, the media gets processed, and the output is streamed directly to the disk. Memory usage is O(1) due to pipelining, and an overly fast uploader is automatically slowed down due to stream backpressure.
|
||||
|
||||
```ts
|
||||
import { ALL_FORMATS, Conversion, FilePathTarget, Input, Mp4OutputFormat, Output, QUALITY_MEDIUM, ReadableStreamSource } from "mediabunny";
|
||||
import { registerMediabunnyServer } from "@mediabunny/server";
|
||||
import { Readable } from "node:stream";
|
||||
import http from "node:http";
|
||||
|
||||
registerMediabunnyServer();
|
||||
|
||||
const server = http.createServer(async (req, res) => {
|
||||
// Read the request body as a stream
|
||||
const stream = Readable.toWeb(req) as ReadableStream<Uint8Array>;
|
||||
const input = new Input({
|
||||
source: new ReadableStreamSource(stream),
|
||||
formats: ALL_FORMATS,
|
||||
});
|
||||
|
||||
// Stream the output directly to the disk, could also stream to S3 etc.
|
||||
const output = new Output({
|
||||
format: new Mp4OutputFormat(),
|
||||
target: new FilePathTarget(`./converted-${crypto.randomUUID()}.mp4`),
|
||||
});
|
||||
|
||||
try {
|
||||
const conversion = await Conversion.init({
|
||||
input,
|
||||
output,
|
||||
video: async track => ({
|
||||
codec: 'avc',
|
||||
height: Math.min(720, await track.getDisplayHeight()),
|
||||
bitrate: QUALITY_MEDIUM,
|
||||
}),
|
||||
});
|
||||
await conversion.execute();
|
||||
|
||||
res.statusCode = 204;
|
||||
res.end();
|
||||
} catch (error) {
|
||||
res.statusCode = 500;
|
||||
res.end();
|
||||
|
||||
console.error("Error processing media:", error);
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(3000);
|
||||
```
|
||||
|
||||
For all the other ways to use Mediabunny, refer to its [guide](https://mediabunny.dev/guide/introduction).
|
||||
|
||||
## Performance
|
||||
|
||||
`@mediabunny/server` is extremely performant as it is a thin wrapper around [NodeAV](https://github.com/seydx/node-av), which itself is a thin wrapper around the FFmpeg C API. All decoders and encoders automatically run on separate threads, keeping the main thread unblocked. Hardware acceleration is automatically detected and utilized on all operating systems whenever available (unless explicitly disabled using `hardwareAcceleration: 'prefer-software'`). Video frame and audio sample data is never copied from FFmpeg unless explicitly requested via `VideoSample.copyTo()` and `AudioSample.copyTo()`, and zero-copy GPU decode -> encode paths are used automatically whenever possible.
|
||||
|
||||
## Advanced usage
|
||||
|
||||
### Usage with NodeAV
|
||||
|
||||
`@mediabunny/server` provides `AvFrameVideoSampleResource` and `AvFrameAudioSampleResource` as a means to create `VideoSample` and `AudioSample` instances that are directly backed by data residing in NodeAV's [`Frame`](https://seydx.github.io/node-av/api/lib/classes/Frame.html) (and therefore FFmpeg's `AVFrame`) without ever having to copy data to or from JavaScript. Reading NodeAV's documentation can help you make full use of this integration.
|
||||
|
||||
To convert between Mediabunny and NodeAV (FFmpeg) worlds, you can do this:
|
||||
```ts
|
||||
import { VideoSample, AudioSample } from 'mediabunny';
|
||||
import { AvFrameVideoSampleResource, AvFrameAudioSampleResource, toAvFrame } from '@mediabunny/server';
|
||||
|
||||
// Frame -> VideoSample
|
||||
new VideoSample(new AvFrameVideoSampleResource(frame), { timestamp });
|
||||
|
||||
// Frame -> AudioSample
|
||||
new AudioSample(new AvFrameAudioSampleResource(frame));
|
||||
// (uses the timestamp in the frame)
|
||||
|
||||
// VideoSample -> Frame
|
||||
await toAvFrame(videoSample, frame);
|
||||
|
||||
// AudioSample -> Frame
|
||||
await toAvFrame(audioSample, frame);
|
||||
```
|
||||
|
||||
#### Electron example
|
||||
|
||||
For example, when using Electron, we may want to capture the app's contents without moving video data from the GPU to the CPU:
|
||||
```ts
|
||||
import { VideoSample } from 'mediabunny';
|
||||
import { AvFrameVideoSampleResource } from '@mediabunny/server';
|
||||
import { HardwareContext, SharedTexture, AV_HWDEVICE_TYPE_VIDEOTOOLBOX } from 'node-av';
|
||||
|
||||
// Create hardware context (platform-specific)
|
||||
const hw = HardwareContext.create(AV_HWDEVICE_TYPE_VIDEOTOOLBOX);
|
||||
using sharedTexture = SharedTexture.create(hw);
|
||||
|
||||
// In Electron paint event with offscreen rendering
|
||||
offscreen.webContents.on('paint', (event) => {
|
||||
const texture = event.texture;
|
||||
if (!texture?.textureInfo) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Import as hardware frame (zero-copy)
|
||||
const frame = sharedTexture.importTexture(texture.textureInfo, { pts: 0n });
|
||||
const sample = new VideoSample(new AvFrameVideoSampleResource(frame), {
|
||||
timestamp: 0,
|
||||
duration: 0,
|
||||
});
|
||||
|
||||
texture.release();
|
||||
});
|
||||
```
|
||||
|
||||
#### Microphone recording example
|
||||
|
||||
Here, we're using NodeAV's Device API to access the user's microphone:
|
||||
```ts
|
||||
import { AudioSample } from 'mediabunny';
|
||||
import { AvFrameAudioSampleResource } from '@mediabunny/server';
|
||||
import { DeviceAPI, Decoder } from 'node-av';
|
||||
|
||||
await using mic = await DeviceAPI.openMicrophone();
|
||||
const audioStream = mic.audio()!;
|
||||
using decoder = await Decoder.create(audioStream);
|
||||
|
||||
let firstTimestamp: number | null = null;
|
||||
for await (const frame of decoder.frames(mic.packets(audioStream.index))) {
|
||||
if (!frame) {
|
||||
break;
|
||||
}
|
||||
|
||||
const sample = new AudioSample(new AvFrameAudioSampleResource(frame));
|
||||
|
||||
if (firstTimestamp === null) {
|
||||
firstTimestamp = sample.timestamp;
|
||||
}
|
||||
|
||||
// Offset timestamps so they start at 0
|
||||
sample.setTimestamp(sample.timestamp - firstTimestamp);
|
||||
|
||||
// Do something with the sample now, like passing it to an AudioSampleSource
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Video and audio processing
|
||||
|
||||
Browser environments ship with many API goodies such as the Canvas 2D API which are a naturally great fit for doing video frame processing, and they integrate well with Mediabunny. On the server, these APIs don't exist, so other approaches must be used:
|
||||
|
||||
#### VideoSample.transform()
|
||||
|
||||
This method allows for simple transformations on `VideoSample` instances and works when `@mediabunny/server` has been registered:
|
||||
```ts
|
||||
const transformed = await sample.transform({
|
||||
width: 640,
|
||||
height: 360,
|
||||
fit: 'cover',
|
||||
});
|
||||
```
|
||||
|
||||
#### NodeAV filter graphs
|
||||
|
||||
FFmpeg's `libavfilter` is an incredibly powerful and generic media processing library, and all of it is directly accessible via NodeAV. It works for both video as well as audio data.
|
||||
|
||||
For example, here we combine Mediabunny's Conversion API with a filter graph to grayscale a video:
|
||||
```ts
|
||||
import { Conversion } from 'mediabunny';
|
||||
import { AvFrameVideoSampleResource, toAvFrame } from '@mediabunny/server';
|
||||
import { Frame, FilterAPI } from 'node-av';
|
||||
|
||||
async function* one(f: Frame) { yield f; }
|
||||
|
||||
const conversion = await Conversion.init({
|
||||
// ...
|
||||
video: {
|
||||
process: async (sample) => {
|
||||
// VideoSample -> Frame
|
||||
using inFrame = new Frame();
|
||||
inFrame.alloc();
|
||||
await toAvFrame(sample, inFrame);
|
||||
|
||||
// Frame -> filter -> AvFrameVideoSampleResource
|
||||
using filter = FilterAPI.create('format=gray');
|
||||
for await (const outFrame of filter.frames(one(inFrame))) {
|
||||
return outFrame && new AvFrameVideoSampleResource(outFrame);
|
||||
}
|
||||
|
||||
return null;
|
||||
},
|
||||
},
|
||||
// ...
|
||||
});
|
||||
await conversion.execute();
|
||||
```
|
||||
|
||||
#### Canvas API polyfills
|
||||
|
||||
Libraries like [Skia Canvas](https://github.com/samizdatco/skia-canvas) provide GPU-enabled polyfills for the Canvas 2D API. Using it with Mediabunny is simply a matter of converting from and to the Canvas API:
|
||||
```ts
|
||||
const width = videoSample.displayWidth;
|
||||
const height = videoSample.displayHeight;
|
||||
|
||||
const canvas = new Canvas(width, height);
|
||||
const ctx = canvas.getContext('2d');
|
||||
|
||||
// Copy data from VideoSample
|
||||
const imageData = ctx.createImageData(width, height);
|
||||
await videoSample.copyTo(imageData.data, { format: 'RGBA' });
|
||||
ctx.putImageData(imageData, 0, 0);
|
||||
|
||||
// Issue draw commands
|
||||
ctx.fillStyle = 'red';
|
||||
ctx.fillRect(20, 20, 100, 60);
|
||||
|
||||
// Convert to VideoSample again
|
||||
const pixels = ctx.getImageData(0, 0, width, height).data;
|
||||
return new VideoSample(pixels, {
|
||||
format: 'RGBA',
|
||||
codedWidth: width,
|
||||
codedHeight: height,
|
||||
timestamp: videoSample.timestamp,
|
||||
duration: videoSample.duration,
|
||||
});
|
||||
```
|
||||
|
||||
## Implementation details
|
||||
|
||||
`@mediabunny/server` uses [NodeAV](https://github.com/seydx/node-av) under the hood which provides N-API C bindings to FFmpeg's C API. Using NodeAV, this package implements [custom decoders and encoders](https://mediabunny.dev/guide/supported-formats-and-codecs#custom-coders) by directly using the APIs provided by `libavcodec`.
|
||||
|
||||
For encoding, video frames and audio samples are transferred to FFmpeg by converting them to an `AVFrame` and are then passed to the correct encoder. The resulting packets are then normalized into the format expected by WebCodecs and the [Mediabunny Codec Registry](https://mediabunny.dev/codec-registry/overview). For decoding, the above process is inverted: packets and decoder metadata are passed to the correct decoder, and the resulting `AVFrame` instances are wrapped in `VideoSample` or `AudioSample` instances. Video frame transformations (resize, rotate, crop) are implemented using the `libavfilter` API.
|
||||
|
||||
Whenever possible, `AVFrame`s are never copied over to JavaScript unless explicitly needed. This enables zero-copy decode -> transformation -> encode paths.
|
||||
|
||||
## License
|
||||
|
||||
`@mediabunny/server` uses the same MPL-2.0 license as Mediabunny.
|
||||
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 136 KiB |
@@ -10,7 +10,7 @@ import { AudioCodec, AudioSample, CustomAudioDecoder, EncodedPacket, type MaybeP
|
||||
import * as NodeAv from 'node-av';
|
||||
import { CODEC_TO_CODEC_ID, getChannelLayout } from './misc';
|
||||
import { assert, toUint8Array } from '../../../src/misc';
|
||||
import { NodeAvFrameAudioSampleResource } from './audio-sample';
|
||||
import { AvFrameAudioSampleResource } from './audio-sample';
|
||||
|
||||
export class NodeAvAudioDecoder extends CustomAudioDecoder {
|
||||
frame!: NodeAv.Frame;
|
||||
@@ -90,7 +90,7 @@ export class NodeAvAudioDecoder extends CustomAudioDecoder {
|
||||
}
|
||||
|
||||
clone.timeBase = new NodeAv.Rational(1, this.config.sampleRate);
|
||||
this.onSample(new AudioSample(new NodeAvFrameAudioSampleResource(clone)));
|
||||
this.onSample(new AudioSample(new AvFrameAudioSampleResource(clone)));
|
||||
}
|
||||
|
||||
async flush(): Promise<void> {
|
||||
|
||||
@@ -15,9 +15,9 @@ import {
|
||||
EncodedPacket,
|
||||
} from 'mediabunny';
|
||||
import * as NodeAv from 'node-av';
|
||||
import { CODEC_TO_CODEC_ID, fromAudioSampleFormat, getChannelLayout } from './misc';
|
||||
import { CODEC_TO_CODEC_ID, getChannelLayout } from './misc';
|
||||
import { assert, toUint8Array } from '../../../src/misc';
|
||||
import { NodeAvFrameAudioSampleResource } from './audio-sample';
|
||||
import { copyAudioSampleToAvFrame, AvFrameAudioSampleResource } from './audio-sample';
|
||||
import {
|
||||
AdtsHeaderTemplate,
|
||||
buildAdtsHeaderTemplate,
|
||||
@@ -133,26 +133,14 @@ export class NodeAvAudioEncoder extends CustomAudioEncoder {
|
||||
|
||||
this.firstExpectedTimestamp ??= audioSample.timestamp;
|
||||
|
||||
if (audioSample._data instanceof NodeAvFrameAudioSampleResource) {
|
||||
if (audioSample._data instanceof AvFrameAudioSampleResource) {
|
||||
this.frame.ref(audioSample._data.frame);
|
||||
} else {
|
||||
// Copy audio data from AudioData to FFmpeg Frame
|
||||
const format = fromAudioSampleFormat(audioSample.format);
|
||||
this.frame.format = format;
|
||||
this.frame.nbSamples = audioSample.numberOfFrames;
|
||||
this.frame.sampleRate = audioSample.sampleRate;
|
||||
this.frame.channelLayout = getChannelLayout(audioSample.numberOfChannels);
|
||||
this.frame.duration = BigInt(Math.round(audioSample.duration * this.config.sampleRate));
|
||||
|
||||
this.frame.allocBuffer();
|
||||
assert(this.frame.data);
|
||||
|
||||
for (let i = 0; i < this.frame.data.length; i++) {
|
||||
audioSample.copyTo(this.frame.data[i]!, { planeIndex: i });
|
||||
}
|
||||
copyAudioSampleToAvFrame(audioSample, this.frame);
|
||||
}
|
||||
|
||||
this.frame.pts = BigInt(Math.round(audioSample.timestamp * this.config.sampleRate));
|
||||
this.frame.duration = BigInt(Math.round(audioSample.duration * this.config.sampleRate));
|
||||
this.frame.timeBase = new NodeAv.Rational(1, this.config.sampleRate);
|
||||
|
||||
const key = `${this.frame.sampleRate}:${this.frame.channels}:${this.frame.format}`;
|
||||
|
||||
@@ -6,9 +6,9 @@
|
||||
* file, You can obtain one at https://mozilla.org/MPL/2.0/.
|
||||
*/
|
||||
|
||||
import { AudioSampleResource } from 'mediabunny';
|
||||
import { AudioSample, AudioSampleResource } from 'mediabunny';
|
||||
import * as NodeAv from 'node-av';
|
||||
import { toAudioSampleFormat } from './misc';
|
||||
import { fromAudioSampleFormat, getChannelLayout, toAudioSampleFormat } from './misc';
|
||||
import { assert, toUint8Array } from '../../../src/misc';
|
||||
|
||||
/**
|
||||
@@ -17,10 +17,13 @@ import { assert, toUint8Array } from '../../../src/misc';
|
||||
* [`AVFrame`](https://ffmpeg.org/doxygen/2.7/structAVFrame.html). You can use this resource to create `AudioSample`
|
||||
* instances that are directly backed by FFmpeg's `AVFrame` without data having to be copied.
|
||||
*
|
||||
* When passed, the `Frame` is now owned by resource, meaning it takes care of closing the frame later. If you want to
|
||||
* keep a copy for your own use, clone the frame first.
|
||||
*
|
||||
* @group \@mediabunny/server
|
||||
* @public
|
||||
*/
|
||||
export class NodeAvFrameAudioSampleResource extends AudioSampleResource {
|
||||
export class AvFrameAudioSampleResource extends AudioSampleResource {
|
||||
/** @internal */
|
||||
_frame: NodeAv.Frame | null;
|
||||
|
||||
@@ -30,7 +33,7 @@ export class NodeAvFrameAudioSampleResource extends AudioSampleResource {
|
||||
*/
|
||||
get frame() {
|
||||
if (!this._frame) {
|
||||
throw new Error('NodeAvFrameAudioSampleResource has been closed.');
|
||||
throw new Error('AvFrameAudioSampleResource has been closed.');
|
||||
}
|
||||
|
||||
return this._frame;
|
||||
@@ -40,7 +43,7 @@ export class NodeAvFrameAudioSampleResource extends AudioSampleResource {
|
||||
super();
|
||||
|
||||
if (frame.getMediaType() !== NodeAv.AVMEDIA_TYPE_AUDIO) {
|
||||
throw new Error('NodeAvFrameAudioSampleResource must be initialized with an audio frame.');
|
||||
throw new Error('AvFrameAudioSampleResource must be initialized with an audio frame.');
|
||||
}
|
||||
|
||||
this._frame = frame;
|
||||
@@ -49,7 +52,8 @@ export class NodeAvFrameAudioSampleResource extends AudioSampleResource {
|
||||
getFormat(): AudioSampleFormat {
|
||||
const result = toAudioSampleFormat(this.frame.format as NodeAv.AVSampleFormat);
|
||||
if (result === null) {
|
||||
throw new TypeError('Unsupported audio sample format: ' + this.frame.format);
|
||||
const name = NodeAv.avGetSampleFmtName(this.frame.format as NodeAv.AVSampleFormat);
|
||||
throw new TypeError(`Unsupported audio sample format: ${name} (${this.frame.format})`);
|
||||
}
|
||||
|
||||
return result;
|
||||
@@ -81,3 +85,17 @@ export class NodeAvFrameAudioSampleResource extends AudioSampleResource {
|
||||
return toUint8Array(this.frame.data[planeIndex]!);
|
||||
}
|
||||
}
|
||||
|
||||
export const copyAudioSampleToAvFrame = (sample: AudioSample, frame: NodeAv.Frame) => {
|
||||
frame.format = fromAudioSampleFormat(sample.format);
|
||||
frame.nbSamples = sample.numberOfFrames;
|
||||
frame.sampleRate = sample.sampleRate;
|
||||
frame.channelLayout = getChannelLayout(sample.numberOfChannels);
|
||||
|
||||
frame.allocBuffer();
|
||||
assert(frame.data);
|
||||
|
||||
for (let i = 0; i < frame.data.length; i++) {
|
||||
sample.copyTo(frame.data[i]!, { planeIndex: i });
|
||||
}
|
||||
};
|
||||
|
||||
@@ -6,13 +6,14 @@
|
||||
* file, You can obtain one at https://mozilla.org/MPL/2.0/.
|
||||
*/
|
||||
|
||||
import { registerDecoder, registerEncoder, registerVideoSampleTransformer } from 'mediabunny';
|
||||
import { AudioSample, registerDecoder, registerEncoder, registerVideoSampleTransformer, VideoSample } from 'mediabunny';
|
||||
import * as NodeAv from 'node-av';
|
||||
import { NodeAvVideoDecoder } from './video-decoder';
|
||||
import { NodeAvVideoEncoder } from './video-encoder';
|
||||
import { NodeAvAudioDecoder } from './audio-decoder';
|
||||
import { NodeAvAudioEncoder } from './audio-encoder';
|
||||
import { transformVideoSample } from './video-sample';
|
||||
import { copyVideoSampleToAvFrame, AvFrameVideoSampleResource, transformVideoSample } from './video-sample';
|
||||
import { copyAudioSampleToAvFrame, AvFrameAudioSampleResource } from './audio-sample';
|
||||
|
||||
const SERVER_LOADED_SYMBOL = Symbol.for('@mediabunny/server loaded');
|
||||
if ((globalThis as Record<symbol, unknown>)[SERVER_LOADED_SYMBOL]) {
|
||||
@@ -59,5 +60,45 @@ export const registerMediabunnyServer = () => {
|
||||
registerVideoSampleTransformer(transformVideoSample);
|
||||
};
|
||||
|
||||
export { NodeAvFrameVideoSampleResource } from './video-sample';
|
||||
export { NodeAvFrameAudioSampleResource } from './audio-sample';
|
||||
export { AvFrameVideoSampleResource } from './video-sample';
|
||||
export { AvFrameAudioSampleResource } from './audio-sample';
|
||||
|
||||
/**
|
||||
* Copies a `VideoSample` or `AudioSample` into the given NodeAV
|
||||
* [`Frame`](https://seydx.github.io/node-av/api/lib/classes/Frame.html), setting up the frame's format, media type,
|
||||
* timing and data. When the sample is already backed by an `AVFrame` (via `AvFrameVideoSampleResource` or
|
||||
* `AvFrameAudioSampleResource`), the frame is ref'd instead of copied for zero-copy reuse.
|
||||
*
|
||||
* For video samples, the frame's time base is always set to 1/1000000 (microsecond accuracy). For audio samples, the
|
||||
* frame's time base is always set to 1/sampleRate.
|
||||
*
|
||||
* @group \@mediabunny/server
|
||||
* @public
|
||||
*/
|
||||
export const toAvFrame = async (sample: VideoSample | AudioSample, frame: NodeAv.Frame) => {
|
||||
if (sample instanceof VideoSample) {
|
||||
if (sample._data instanceof AvFrameVideoSampleResource) {
|
||||
frame.ref(sample._data.frame);
|
||||
} else {
|
||||
if (sample.format === null) {
|
||||
throw new Error('Cannot convert foreign VideoSample with unknown (null) format.');
|
||||
}
|
||||
|
||||
await copyVideoSampleToAvFrame(sample, frame, null);
|
||||
}
|
||||
|
||||
frame.pts = BigInt(sample.microsecondTimestamp);
|
||||
frame.duration = BigInt(sample.microsecondDuration);
|
||||
frame.timeBase = new NodeAv.Rational(1, 1e6);
|
||||
} else {
|
||||
if (sample._data instanceof AvFrameAudioSampleResource) {
|
||||
frame.ref(sample._data.frame);
|
||||
} else {
|
||||
copyAudioSampleToAvFrame(sample, frame);
|
||||
}
|
||||
|
||||
frame.timeBase = new NodeAv.Rational(1, sample.sampleRate);
|
||||
frame.pts = BigInt(Math.round(sample.timestamp * sample.sampleRate));
|
||||
frame.duration = BigInt(sample.numberOfFrames);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -10,7 +10,7 @@ import { CustomVideoDecoder, VideoCodec, EncodedPacket, VideoSample, type MaybeP
|
||||
import * as NodeAv from 'node-av';
|
||||
import { CODEC_TO_CODEC_ID, getHardwareDecoderCodec, LIBVPX_VP9 } from './misc';
|
||||
import { assert, binarySearchLessOrEqual, simplifyRational, toUint8Array } from '../../../src/misc';
|
||||
import { NodeAvFrameVideoSampleResource } from './video-sample';
|
||||
import { AvFrameVideoSampleResource } from './video-sample';
|
||||
|
||||
export class NodeAvVideoDecoder extends CustomVideoDecoder {
|
||||
frame!: NodeAv.Frame;
|
||||
@@ -187,7 +187,7 @@ export class NodeAvVideoDecoder extends CustomVideoDecoder {
|
||||
throw new Error('Frame clone allocation failed.');
|
||||
}
|
||||
|
||||
this.onSample(new VideoSample(new NodeAvFrameVideoSampleResource(clone), {
|
||||
this.onSample(new VideoSample(new AvFrameVideoSampleResource(clone), {
|
||||
timestamp,
|
||||
duration,
|
||||
}));
|
||||
|
||||
@@ -24,7 +24,7 @@ import {
|
||||
unmapMatrixCoefficients,
|
||||
unmapTransferCharacteristics,
|
||||
} from './misc';
|
||||
import { copyVideoSampleToAvFrame, NodeAvFrameVideoSampleResource } from './video-sample';
|
||||
import { copyVideoSampleToAvFrame, AvFrameVideoSampleResource } from './video-sample';
|
||||
import {
|
||||
AvcNalUnitType,
|
||||
extractAv1CodecInfoFromPacket,
|
||||
@@ -187,7 +187,7 @@ export class NodeAvVideoEncoder extends CustomVideoEncoder {
|
||||
assert(this.codecContext);
|
||||
}
|
||||
|
||||
if (videoSample._data instanceof NodeAvFrameVideoSampleResource) {
|
||||
if (videoSample._data instanceof AvFrameVideoSampleResource) {
|
||||
this.frame.ref(videoSample._data.frame);
|
||||
} else {
|
||||
if (videoSample.format === null) {
|
||||
|
||||
@@ -48,10 +48,13 @@ const JPEG_RANGE_PIX_FORMATS = new Set([
|
||||
* When using Electron, you can directly create `Frame` instances without the data having to leave the GPU. For more,
|
||||
* see [NodeAV's docs](https://seydx.github.io/node-av/api/lib/classes/Frame.html).
|
||||
*
|
||||
* When passed, the `Frame` is now owned by resource, meaning it takes care of closing the frame later. If you want to
|
||||
* keep a copy for your own use, clone the frame first.
|
||||
*
|
||||
* @group \@mediabunny/server
|
||||
* @public
|
||||
*/
|
||||
export class NodeAvFrameVideoSampleResource extends VideoSampleResource {
|
||||
export class AvFrameVideoSampleResource extends VideoSampleResource {
|
||||
/** @internal */
|
||||
_frame: NodeAv.Frame | null;
|
||||
|
||||
@@ -61,7 +64,7 @@ export class NodeAvFrameVideoSampleResource extends VideoSampleResource {
|
||||
*/
|
||||
get frame() {
|
||||
if (!this._frame) {
|
||||
throw new Error('NodeAvFrameVideoSampleResource has been closed.');
|
||||
throw new Error('AvFrameVideoSampleResource has been closed.');
|
||||
}
|
||||
|
||||
return this._frame;
|
||||
@@ -71,7 +74,7 @@ export class NodeAvFrameVideoSampleResource extends VideoSampleResource {
|
||||
super();
|
||||
|
||||
if (frame.getMediaType() !== NodeAv.AVMEDIA_TYPE_VIDEO) {
|
||||
throw new Error('NodeAvFrameVideoSampleResource must be initialized with a video frame.');
|
||||
throw new Error('AvFrameVideoSampleResource must be initialized with a video frame.');
|
||||
}
|
||||
|
||||
this._frame = frame;
|
||||
@@ -169,7 +172,7 @@ export class NodeAvFrameVideoSampleResource extends VideoSampleResource {
|
||||
|
||||
dstFrame.sampleAspectRatio = srcFrame.sampleAspectRatio;
|
||||
|
||||
return new VideoSample(new NodeAvFrameVideoSampleResource(dstFrame), init);
|
||||
return new VideoSample(new AvFrameVideoSampleResource(dstFrame), init);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -213,7 +216,7 @@ export const transformVideoSample = async (
|
||||
let srcFrame: NodeAv.Frame;
|
||||
let srcFrameOwned = false;
|
||||
|
||||
if (sample._data instanceof NodeAvFrameVideoSampleResource) {
|
||||
if (sample._data instanceof AvFrameVideoSampleResource) {
|
||||
srcFrame = sample._data.frame;
|
||||
} else {
|
||||
if (sample.format === null) {
|
||||
@@ -295,7 +298,7 @@ export const transformVideoSample = async (
|
||||
const getRet = await bufferSink.buffersinkGetFrame(dstFrame);
|
||||
NodeAv.FFmpegError.throwIfError(getRet, 'buffersinkGetFrame');
|
||||
|
||||
return new VideoSample(new NodeAvFrameVideoSampleResource(dstFrame), {
|
||||
return new VideoSample(new AvFrameVideoSampleResource(dstFrame), {
|
||||
timestamp: sample.timestamp,
|
||||
duration: sample.duration,
|
||||
rotation: 0, // baked in by the filter graph
|
||||
|
||||
Reference in New Issue
Block a user