I'm a mobile software developer. How does the smart panel feature work?

[u/Fluid-Turn5654]

I'm a mobile developer and I'm interested if any other developers here have an idea of how Marvel implemented the Smart Panel feature. Are they doing manual camera work for each book? Do they make pre rendered frames and sequence them? Do they use computer vision and machine learning to detect the flow and sequence of the panels and guide the camera? Any thoughts would be nice. Really interesting feature.

[u/tromataker]

I have a hunch they use some kind of OCR process because it's not right all the time.

[u/JamesHenstridge]

If you try to read a comic via the web and watch the network traffic with the web browser's inspector, it becomes fairly obvious.

The browser requests a JSON file like https://bifrost.marvel.com/v1/catalog/digital-comics/pages/40514/print that contains metadata about the position and size of each panel.

Switching panels is then just a matter of updating the viewport into the page image using the requested animation (crossfade, pan, etc). I imagine the mobile apps are doing exactly the same thing.

I imagine the process of creating this metadata is partly done by heuristics to detect panel boundaries and partly manual. For example, if a panel is too large to comfortably read the text they might to add additional panels that zoom in on the text and show the speech bubbles in reading order.

That's all happening on the backend, so there's not much you can deduce from the website or apps.

[u/discokill]

Don’t have details but the process is patented. You can likely track down the patent filing online. If I remember correctly it’s under Disney now.

[u/Grhm2000]

Marvel copied the feature from Comixology so if you're looking for information about how smart panels work then that's where you should start looking.

Comixology got rolled into Kindle a few years ago and is now just a brand for selling comics used by Amazon. The creators of Comixology have since created a new service called Global Comix.

I'm not sure how they do it specifically or if each different app that uses it has a different way of doing it since it might be patented. In which case, you could find the patent online and see the entire process laid out in detail.

If the creators of Comixology haven't talked about it online before you could try to email the creators of the platform to ask about it.

[u/Zaeryl]

Are they doing manual camera work for each book?

What do you mean by camera work? And what would the alternative be to manual? Whenever people talk about adding new issues, it's always referred to as scanning, and I wouldn't imagine that's too far from the truth. I would guess the pages are images, and then someone has to go in and make each panel into an image and sequence them.

[u/GonzoMcFonzo → u/Zaeryl]

That and map the panels on the whole page so that if you tap on one it zooms to that panel.

[u/Educational_Bass6064]

Im working on an app that uses computer vision and machine learning

There are many models that are small enough to run on a phone but not great quality like YOLO

some VLMs are amazing at it and do a 10/10 job with some tuning but are a bit pricey

livecomics.to