Core-SDK

Core-SDK is the SDK for the canvas rendering part, which can draw project data (JSON) onto the Canvas. It can be used for secondary development to create video-related mini-programs, H5, PC video editors, and other web-based online tools.

github demo

Usage:

import { VideoCoreSDK } from "@h5/videoCoreSDK";

const vc = new VideoCoreSDK({
  data: MovieData, // Required, project data
  workerPath: "assets", // Optional, reference directory for decode.worker.js, default is assets
  registerId: "", // Required, registration ID, bound to the domain name
  movieId: "movieID", // Unique identifier within the instance, cannot be the same for multiple instances
  EModuleEffectSourcePath: "/assets/effectcanvas/", // Effect resource module loading path
  resourceHost: "https://cdn.h5ds.com", // Host for resource loading
  scale: 1, // Optional, canvas scaling ratio, default is 1
  env: "preview", // Required, rendering mode: editor (edit mode, draggable for editing); preview (preview mode, can only preview, less memory consumption)
  currentTime: 0, // Optional, default start time is 0
  plugins: [], // Optional, extension plugins
});

const core = await vc.init();

/**
 * Assuming the project data is movieData, modify data -> update view
 */
// 1. Get the first element in the project data
const [element] = movieData.elements;

// 2. Modify the coordinates of the first element to [100, 200]
element.style.x = 100;
element.style.y = 200;

// 3. Update the view
core.update();

You can get the vc instance through new VideoCore(...), then get core through vc.init(). The core instance contains many properties and methods, which are introduced below one by one along with usage tips.

// VideoCoreSDK parameter description typescript
export interface IMovieProps {
  data: types.MovieData; // Project data
  workerPath?: string; // Worker address
  registerId?: string; // Registration ID
  movieId?: string; // Unique identifier within the instance, cannot be the same for multiple instances
  EModuleEffectSourcePath?: string; // Effect loading path
  scale?: number; // Canvas scaling ratio, default is 1
  resourceHost?: string; // Host for resource loading
  stopControl?: boolean; // Stop controller, editor uses controller by default, can disable controller in editor through this parameter
  useRecordManager?: boolean; // Use history records, history records are enabled by default
  plugins?: types.PluginConfig[]; // Extension plugins
  target?: HTMLElement | null; // DOM container where canvas is placed, default is body
  times?: [number, number]; // If this parameter is passed, only resources in this interval will be loaded, used with export
  env: types.Env; // Controlled by environment variables, env:  'editor' | 'preview' | 'export';
  server?: boolean; // Set to true for server-side rendering, default is false
  // Must preload resources when exporting MP4, no need to preload resources in the editor, so there is a parameter here to control whether resources need to be preloaded
  fetchSourceBeforeRender?: boolean; // Whether to cache resources before rendering, default is false
  currentTime?: number; // Time, can be used in editor and preivew, need to call update method in export
  // This method accepts a method to change currentTime, triggers currentTime change
  triggerCurrentTime?: (t: number) => void;
  // Select multiple elements in edit mode
  onSelectElements?: (elementIds: string[]) => void;
  // Execute after controller changes
  onControlChangeEnd?: (elementIds: string[]) => void;
  // Resource loading progress
  onSourceProgress?: (p: number) => void;
  // Single resource loading progress
  onEachSourceProgress?: (n: { src: string; p: number; total: number }) => void;
  // Pause event
  onPause?: () => void;
  // Execute after initialization succeeds
  initSuccess?: () => void;
  // Execute after rendering
  callback?: (c: Store) => void;
}

Property Description

Only the properties of the instance core are listed below, with the data type after the colon. Private properties all start with _.

.target?: HTMLElement | null;

DOM container where canvas is placed, default is body

.movieId: string

Unique identifier of the instance, can be a random string, default is 'movieId'

.workerPath: string

Worker address, default is /assets/worker/decode.worker.js, this worker is needed to decode video frames when exporting video

.currentTime: number

Current display time

.times: [number, number] | null

When exporting, you can pass times to indicate exporting a video for a specified period of time, in seconds

.controlModeType: 'editElement' | 'editMask'

Controller mode, divided into element edit mode and mask edit mode, only mask can be edited in mask edit mode

.bodyContainer: PIXI.Container

Body container, used to store PIXI elements

.captionContainer: PIXI.Container

Caption container, used to store caption elements

.data: types.MovieData

Animation JSON data

.events: events.EventsName

Instance of event names, events can be called by triggering event names

// Update view
pubsub.publish(core.events.UPDATE_STAGE);

// Event names and descriptions:
events.UPDATE_STAGE; // Update view
events.VISIBLE_CONTROL; // Controller show/hide
events.STOP_GROUP_DRAG; // Disable group selection
events.SORT_CONTAINER; // Reorder container
events.UPDATE_CONTROL_TARGET; // Update controller after target size changes
events.UPDATE_CONTROL_LIST; // Controller update list
events.TRIGGER_CONTROL; // Trigger controller to select element
events.TRIGGER_CONTROL_NO_CALLBACK; // Only trigger controller, do not call onSelected callback
events.UPDATE_MOVIE; // Update Movie.tsx
events.CONTROL_CHANGE_END; // Controller end
events.CONTROL_CHANGE_START; // Controller start

.record: RecordObject

Collection of history record methods, this parameter is only available when env is editor

export interface RecordItem {
  desc: string; // Description information
}

export interface RecordObject {
  add: (item: RecordItem) => void; // Add history record
  debounceAdd: (item: RecordItem) => void; // Add history record using debounce function
  redo: () => void;
  undo: () => void;
  manager: RecordManager; // Data management for history records
}

.resourceManage: ResourceManage

Resource manager instance, please refer to the class: packages\video-core\src\react-pixi\manage\resourceManage.ts

.movieSize: { width: number; height: number }

Video size

.app: Application | undefined

pixijs instance

.playing: boolean

Whether in playback mode, not in playback by default

.env: 'editor' | 'preview' | 'export'

Current operating environment, editor means in editor mode, preview means in preview mode, export means in export mode

.groups: types.BaseElement[][]

Data grouped by trackIndex

.zIndexObjects: Record<string, number>

Record element zIndex data for controller element usage

.cacheControlElementIds: string[]

Array of element IDs recorded by the controller

.usedMediaInfo: Record<string, types.UsedMediaItem>

Cache all media resources video & audio for audio synthesis

.elementReadyMark: Record<string, 'start' | 'success' | 'error'>

Record layer preparation success, used to handle asynchronous issues in react, tring can be elementId or other parameters

.renderAsyncMark: Record<string, 'start' | 'success' | 'error'>

Reference format: {elementId_time: 'start' | 'success' | 'error'}

The rendering process is asynchronous, used to detect whether rendering is successful. Before rendering, put the element's id and time into the object, after successful rendering, the time is null. If all rendering is completed, the mark should be {}, and if it hasn't been rendered after 1 second, no processing will be done. Therefore, to ensure performance issues, the rendering time of each frame must be controlled within 1 second

.controlElements: ControlChangeValues[]

Record data after controller changes

export interface ControlChangeValues {
  elementId: string; // Target element ID
  x: number;
  y: number;
  alpha?: number;
  width: number;
  height: number;
  rotation: number;
}

Method Description

get hideLockData(): number

Hide and lock are for the entire track, return lock, hide show element records, this parameter should be set when calculating group

return Record<number, { hide: boolean; lock: boolean }>;

getPixiContainerById(ids: string[]): PIXI.Container[]

Get pixi container objects by ids

pixiContainerToImageById(id: string): Promise<HTMLImageElement>

Find pixi container by id, then export as image

initHideLock(hideLock: Record<number, { hide: boolean; lock: boolean }>)

The hide and lock information of tracks in the timeline will be recorded in movieData._hideLock ={}, which needs to be initialized, the key in _hideLock is trackIndex

getHideLock(trackIndex: number): { hide: boolean;lock: boolean;}

Get track hide and lock parameters

setHideLock(params: { trackIndex: number; hide?: boolean; lock?: boolean }, updateMovie?: boolean)

Set hide and lock for the specified track (trackIndex), the second parameter indicates whether to update movie

changeControlMode(modeType: 'editElement' | 'editMask', id: string)

Switch edit mode, which are element edit and mask edit respectively.

getFrameItem(elementData: types.BaseElement): types.FrameItem | null

Get current frame data, if there is data at the current time node, return the corresponding frame data (error 0.1 seconds)

asyncFrameAnimateStatus(elementData: types.BaseElement, status: types.FrameItem | null)

Synchronize frame animation data with element status, can be understood as updating element display status through frame data

getFrameStatusByCurrentTime(elementData: types.BaseElement, currentTime?: number): types.FrameItem | null

Get frame status at specified time position, if the second parameter is not passed, use the time at the cursor position by default

getFrameStatus(elementData: types.BaseElement, relativeTime: number | null): types.FrameItem | null

Get frame status at specified time position

removeTrackMedias(trackIndex: number, resourceId: string)

Remove media data, called when deleting elements

checkSupportTransition(elementId: string): boolean

Determine whether the element supports inserting transition data, transitions can only be inserted between two adjacent video or image elements

removeInvalidTransition()

Remove invalid transition animations

play()

Play video

pause()

Pause video

sortZIndexNewData()

Reset zindex when saving, return new data, the reason for resetting zindex parameter is that when inserting new tracks, the subscript may be a decimal, reset will update it to integer

step(time: number)

Set video time

runAnimate()

Execute play operation, fps is 32 frames by default, the frequency of requestAnimationFrame may be 60fps, need to play through FPS. The reason for not using setTimeout is that setTimeout will cause large errors and cannot correct the time

updateControl(type: 'updateTarget' | 'updateList' | 'trigger' | 'triggerNoCallback' | 'visible' | 'stopGroupDrag', doms?: HTMLElement[] | SVGElement[] | string[] | { visible: boolean; elementId: string } | boolean)

Method to update the controller

updateTarget: Update controller after target size changes

updateList: Update controller after targets list changes

trigger: Trigger element to be selected

triggerNoCallback: Only trigger controller, do not call onSelected callback

visible: Show/hide controller { visible, elementId }

stopGroupDrag: Disable & enable group selection

// Trigger controller to select elementId1
movie.updateControl("trigger", ["elementId1"]);

updateSortTracks()

Track sort update, need to call this method again to update the z-index issue in pixi

getTotalTime()

Get total duration of video

getElementDataByIds(ids: string[])

Get elements by ids

getElementDataByTypes(types: types.ElementType[], data?: types.MovieData): types.BaseElement[]

Get elements by type

getCloneData()

Get clone data of data, recalculate trackIndex to make it integer, starting from 1

addResource(url: string, styleSize?: types.StyleSize): Promise<types.Resource>

Add resource, resource will be synchronously added to resourceManage and cached

export interface StyleSize {
  width: number;
  height: number;
}

addElementByResource(resource: types.Resource, params: { elementType: types.ElementType; time: number; trackIndex?: number; duration?: number })

Add element through resource data

core.addElementByResource(resource, {
  elementType: "image", // Insert image
  time: 10, // Start from 10 seconds
  trackIndex: 2, // Insert into the second track
  duration: 5, // Display for 5 seconds
});

addElementNoSource( info: Record<string | 'attrs', any>, params: { elementType: types.ElementType; time: number; trackIndex?: number; duration?: number })

Add elements without resource, such as: text, effect, cutScene, etc.

update(time?: number)

Update Movie component, for example, if the _dirty parameter of an element is modified, this function can be called, or if the data structure is modified, this method can also be called. In export environment, use update to update component, because in the export process, rendering is an asynchronous process, need to judge whether rendering is successful

capture(): string

Capture current view screen, return base64 image

controlChangeStart(elementIds: string[])

This method is triggered when the controller is selected, no need to call it separately

controlChangeEnd(elementIds: string[])

Callback function when controller ends, no need to call it separately

destroy()

Destroy instance, memory recycling

Event Related

onControlChangeEnd: ((elementIds: string[]) => void)

Bind event for end of controller changes

onControlChangeStart: ((elementIds: string[]) => void)

Bind event for start of controller changes

onEachSourceProgress: (n: { src: string; p: number; total: number }) => void

This event is triggered for each resource loading progress

triggerCurrentTime: (t: number) => void

Event that triggers currentTime change

triggerRecordSelectElements: (ids: string[]) => void

Trigger element selection

React DEMO

import { useEffect, useRef, useState } from 'react';
import './App.less';

import { VideoCoreSDK } from '@video/core/src/react-pixi/CoreSDK.tsx';
import { data } from './data';
import { Slider } from '@douyinfe/semi-ui';

function App() {
  const videoCoreRef = useRef<any>(null);
  const [currentTime, setCurrentTime] = useState(0);
  const [duration, setDuration] = useState(0);

  useEffect(() => {
    // Ensure only one instance is created
    if (!videoCoreRef.current) {
      const vc = new VideoCoreSDK({
        data: data, // Required, project data
        target: document.getElementById('video-container')!, // Required, rendering target
        movieId: 'movieID', // Unique identifier within the instance, cannot be the same for multiple instances
        env: 'preview', // Required, rendering mode: editor (edit mode, draggable for editing); preview (preview mode, can only preview, less memory consumption)
        registerId: 'H5DS', // Required, registration ID, bound to the domain name
        resourceHost: 'https://cdn.h5ds.com',
        EModuleEffectSourcePath: '/assets/effectcanvas/', // Effect resource module loading path
        workerPath: 'assets', // Optional, reference directory for decode.worker.js, default is assets
        scale: 1, // Optional, canvas scaling ratio, default is 1
        currentTime, // Optional, default start time is 0
        plugins: [], // Optional, extension plugins
        triggerCurrentTime: (currentTime: number) => {
          setCurrentTime(currentTime);
        },
      });
      vc.init().then((core: any) => {
        videoCoreRef.current = core;
        setDuration(core.getTotalTime());
      });
    }

    // Cleanup function, destroy instance when component unmounts
    return () => {
      if (videoCoreRef.current) {
        // videoCoreRef.current.destroy();
        videoCoreRef.current = null;
      }
    };
  }, []);

  return (
    <div
      style={{
        width: '960px',
      }}
    >
      <div id="video-container"></div>
      <div>
        {!!duration && (
          <Slider
            value={currentTime}
            max={duration}
            step={0.01}
            onChange={value => videoCoreRef.current?.triggerCurrentTime(value)}
          />
        )}
      </div>
      <div className="list">
        <a
          onClick={() => {
            videoCoreRef.current?.play();
          }}
        >
          play
        </a>
        <a
          onClick={() => {
            videoCoreRef.current?.pause();
          }}
        >
          pause
        </a>
      </div>
    </div>
  );
}

export default App;