On-device Visual Perception for Light-weight Browser Agents

Source summary

Build a browser vision-agent prototype with local screen perception and sensitive-data redaction before any context leaves the client. Send only sanitised context to a server LLM/VLM, then execute returned actions or process its response locally. Demonstrate Chrome/Firefox extension/JS and server integration end to end, balancing visual accuracy, PII recall/precision, redaction, resources and latency. Use an offline-deployable open model server-side; hosted use is allowed during SIH. Privacy filtering is mandatory rather than optional.