Unveiling the Vision Transformer: A Leap in Video Generation The closest open-source model to SORA is Latte, which uses the same Vision Transformer architecture. So, what makes the Vision Transformer so outstanding, and how does it differ from previous methods? You can Train Your Own SORA Model. Latte hasn’t open-sourced its text-to-video training code. We’ve replicated this code from the paper and made it available for anyone to use in training their own SORA alternative model. Let’s discuss how effective our training was. From 3D U-Net to Vision Transformer Image generation has advanced significantly, with the U-Net model structure being the most commonly used: If you’re confused about the network structures, remember the key principle of deep learning: “Just Add More Layers!” Vision Transformer: A Game Changer In 3D U-Net, the transformer can only function within the U-Net, limiting its view. The Vision Transformer, however, enables transformers to globally manage video generation. Training Your Open-Source SORA Alternative with Latte Latte uses the video slicing sequence and Vision Transformer method discussed. While Latte hasn’t open-sourced its text-to-video model training code, we’ve replicated it here: GitHub Repo. Training involves three steps: For more details, see the GitHub repo. They’ve also made improvements to the training process: Model Performance The official Latte video shows impressive performance, especially in handling significant motion. However, our own tests indicate that while Latte performs well, it isn’t the top-performing model. Other open-source models have shown better performance. We will continue to share information on models with better performance, so stay tuned to Tectonic’s Insights. Hardware Requirements Due to its large scale, training Latte requires an A100 or H100 with 80GB of memory. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more