

The system operates in two phases, as shown in Figure~\ref{fig:workflow}. First, we create a spatial knowledge graph for each building during the Knowledge Construction phase and store it in the RAG Knowledge base. During the Navigation phase, we retrieve a multi-tier context for a user’s query and produce step-by-step directions for navigating a building. Each floor plan image is treated as input to the complete agentic pipeline, implemented as a multi-agent structure: Parser $\rightarrow$ Graph Builder $\rightarrow$ Self-Critic $\rightarrow$ Planner $\rightarrow$ Safety Evaluator, while ingesting the corresponding RAG. To gain insight into the system's efficacy, we conduct a trial in which a sighted evaluator physically walked each route in the corresponding UMBC building(s) while listening to the generated instructions.


For each floor plan,  a knowledge graph is constructed in which each floor plan is represented as $\G = (\V, \E, \A)$, where $\V$ is the set of room nodes, $E \subseteq \V \times \V$ is the set of edges where each edge is mediated by a door or structural feature between two rooms in $\V$, and $A \in \{0,1\}^{N \times N}$ is a symmetric adjacency matrix, where $A[i][j] = 1$ if two rooms, $v_i$ and $v_j$, share a door or passage and $0$ otherwise. Each node $v$ in the graph contains the room name and semantic type (e.g., room, hallway, elevator, or stairs) as well as the centroid coordinates of the room obtained via OCR. We also store the estimated dimensions of each room along with an estimate of the OCR confidence. Each edge $E$ contains the structural element that mediates the edge (e.g., door) along with a bounding box around that element and an estimate for the traversal cost of traversing through the edge.

%--------------------------------------------------------------------
\subsection{Agentic Pipeline}\label{sec:agentic}

The agentic pipeline decomposes floor plan understanding into sequential agents, that managed by a central controller with retry logic, as shown in Figure~\ref{fig:workflow} (left) and Algorithm~\ref{alg:orchestrator}. The specific prompts used by each agent are detailed in the Appendix.

\paragraph{Agent~1: Parser.}
For inputting a floor plan image $I$ and a navigation query $(s, d)$, the Parser generates an initial graph estimate $\hat{\G}_0$ by integrating the image with object detection $\mathcal{D}$ with YOLOv12 as an object detection with class names, confidences and bounding boxes, and outputs the detected rooms, a symmetric relation matrix, and the information of the detected rooms. By incorporating detection positions into grounding context, the Parser avoids generating more adjacencies that are generated by LLMs.

\paragraph{Agent~2: Graph Builder.}
The Graph Builder processes the Parser output into a validated node-edge graph. For each detected door, the two nearest rooms, determined by pixel centroid distance, are linked. The entries which are not detected are preserved as a passage edge, and an adjacency matrix is rebuilt from the edge list to make sure both sides agree.

\paragraph{Agent 3: Self-Critic}
The Self-Critic enforces five structural checks: Breadth-First Search (BFS) connectivity, door-edge consistency, and spatial coherence. It flags centroid distances beyond $\mu + 2\sigma$. It also checks matrix symmetry and isolated nodes. Failures trigger the Orchestrator to re-invoke the Parser with the appended error context, up to $R_c = 2$ retries, as shown in Figure~\ref{fig:workflow}.
This mechanism, inspired by Self-Refine~\cite{madaan2023selfrefine}, enhances graph extraction accuracy and produces reliable navigation instructions.

\paragraph{Agent~4: Planner.}
The Planner computes the shortest path $P^* = \textrm{BFS}(\A, s, d)$ and generates heading-tracked, turn-by-turn instructions that incorporate landmark references from the detection data. Any existing safety annotations are included in the planning prompt.

\paragraph{Agent~5: Safety Evaluator.}
The Safety Evaluator applies four rule-based checks, including narrow passages (clearances less than 90 cm between walls and objects), missing door-edge flags, long traversals (traversing more than 5 steps or routing on a place with fewer landmarks, dead ends, and checking instructions that can the user to hit the wall, which can stem from hallucinating the LLMs. These hazards are potential safety concerns for visual grounding models that directly process a floor plan image together with natural language, with no preprocessing steps.  Each hazard is assigned a severity score $\sigma \in [1,5]$ along with a mitigation recommendation. If a hazard receives $\sigma \geq 4$, the system initiates a re-routing feedback loop to the Planner, as shown in Figure~\ref{fig:workflow}.

\begin{algorithm}
\caption{Agentic Pipeline for Floor Plan Parsing}
\begin{algorithmic}[1]
\Require Floor plan image $I$, query $(s, d)$, max retries $R_c$
\Ensure Navigation steps $\mathcal{N}$

\State $\mathcal{D} \leftarrow \textrm{YOLOv12}(I)$ \Comment{Object detection}

\For{$r = 0$ \textbf{to} $R_c$}
    \State $\hat{\G}_0 \leftarrow \textrm{Parser}(I,\,\mathcal{D})$
    \State $\G \leftarrow \textrm{GraphBuilder}(\hat{\G}_0,\,\mathcal{D})$
    \State $\mathit{passed} \leftarrow \textrm{SelfCritic}(\G,\,\mathcal{D})$
    \If{$\mathit{passed}$}
        \State \textbf{break}
    \EndIf
\EndFor

\State Ingest $\G$ and $\mathcal{D}$ into Knowledge Base

\State $P^* \leftarrow \textrm{BFS}(\A,\,s,\,d)$
\State $\mathcal{N} \leftarrow \textrm{Planner}(\G,\,P^*,\,s,\,d)$

\State $\sigma_{\max} \leftarrow \textrm{SafetyEval}(\G,\,\mathcal{N})$

\If{$\sigma_{\max} \geq 4$}
    \State $\mathcal{N} \leftarrow \textrm{Planner}(\G,\,P^*,\,s,\,d,\,\sigma_{\max})$
    \Comment{Re-route}
\EndIf

\State \Return $\mathcal{N}$
\end{algorithmic}
\label{alg:orchestrator}
\end{algorithm}


\subsection{Preprocessing}\label{sec:preproc}

Localizing room labels on the floor plans is difficult due to variable font styles, orientations, and background clutter. To address this, we adopt a multi-stage detection strategy that falls back via several methods until labels are successfully located. Specifically, we apply multi-engine OCR (e.g., PP-OCR~\cite{du2020ppocr}, EasyOCR, Tesseract) with fuzzy matching (Levenshtein ratio $\geq 0.55$) against known labels. We employ YOLOv12~\cite{tian2025yolov12}, fine-tuned to detect floor plan elements, including doors, walls, and windows. Each detection produces a node containing the class label, confidence score, bounding box, and center coordinates. These detections serve dual roles: grounding the parser’s LLM prompt and enabling the Graph Builder’s spatial door–room association.

%--------------------------------------------------------------------
\subsection{RAG Knowledge Base}\label{sec:rag}

After knowledge construction, the validated graph $\G$ and associated metadata are ingested into a persistent, queryable multimodal knowledge base, as depicted in the central panel of Figure~\ref{fig:workflow}.

\textbf{Graph-based Relational Knowledge:} The adjacency matrix and room metadata are loaded into a NetworkX~\cite{hagberg2008networkx} graph. Edge attributes include the mediating element, bounding box, and spatial descriptions for each node. Path queries use BFS, guaranteeing shortest paths in $O(|\V| + |\E|)$, providing the structural backbone for physically valid routes.

\textbf{Vector-based Semantic Knowledge:} For semantic embeddings, a rich text document is constructed for each room, door, and room-to-room transition, incorporating room name and type, dimensions, OCR confidence, neighboring rooms, accessible doors with VLM-generated context (wall side, spatial relationships), landmark descriptions, and surface materials. Documents are embedded and indexed in ChromaDB~\cite{chroma2024} for semantic retrieval:
$\textrm{doc}_i^* = \arg\max_j \textrm{sim}(\mathbf{q}, \mathbf{e}_j)$.
This modality transforms generic instructions (e.g., ``go to hallway'') into precise guidance (e.g., ``exit through Door~D2 on the south wall; the hallway has a smooth tile floor'').

\textbf{Visual Grounding Context:} The original image path, all YOLOv12 detections, and OCR label positions are stored. A VLM spatial analysis module generates context for each element (connected rooms, wall locations, adjacency), fusing visual grounding with the graph and vector modalities. Upon a query, the navigator retrieves: (1)~the shortest path from the graph store, (2)~rich semantic documents, and (3)~detection summaries and visual context. These are assembled into a structured prompt that guides the LLM along the exact graph path with landmark and sensory details. A post-generation cross-validation verifies that all referenced rooms exist on the computed path $P^*$; hallucinations or impossible transitions trigger instruction regeneration.

\subsection{Localization}
During navigation, fiducial markers serve as checkpoints. When a specific ArUco marker referenced in the navigation instructions is scanned, the system confirms the instruction. Otherwise, it sends an alert and reroutes the user from the detected checkpoint to the destination. Figure~\ref{fig:three-tier} shows a concrete example of the hybrid knowledge base populated from a CVC-FP floor plan dataset~\cite{CVC_FP}, illustrating the complementary information contributed by the graph, vector, and visual tiers. We conduct 101 navigation trials per route type, including short, medium, and long, to assess end-to-end system performance.

% \begin{figure}[!t]
%     \centering
%     \includegraphics[width=\columnwidth]{FP-1.png}

%   \caption{Schematic Structure of proposed Agentic system.}
% \end{figure} \label{fig:CVC-FP}


\begin{figure}[t]
  \centering
  \begin{tcolorbox}[colback=blue!3!white, colframe=blue!60!black,
    title=\textbf{Multimodal Knowledge Base (KB)},
    fonttitle=\sffamily\small, boxrule=0.5pt, left=4pt, right=4pt, top=4pt, bottom=4pt]
    
  {\scriptsize \textbf{Graph-based Relational Knowledge}}
  {\scriptsize
  \begin{verbatim}
Transition: Cuisine -> Repas
Via: door | Door ID: Door_D3
Door position (bbox): [160, 260, 180, 280]
Door description: Door_D3 to Repas
Cuisine size: 11.85 m2 (3.5 m x 3.4 m)
Repas size: 38.22 m2 (5.5 m x 6.95 m)
...
  \end{verbatim}
  }
  
  \tcbline
  
  {\scriptsize \textbf{Vector-based Semantic Knowledge}}
  {\scriptsize
  \begin{verbatim}
Room: Cuisine
Type: room | Size: 11.85 m2
Dimensions: 3.5 m x 3.4 m
Doors (3): Door_D2 to Cellier; Door_D3 to Repas;
           Door_D6 to Hall
Windows (1): West wall
Connected rooms: Cellier to North; Repas to South;
                 Hall to East
                 ....
  \end{verbatim}
  }
  
  \tcbline
  
  {\scriptsize \textbf{Visual Grounding Context}}
  {\scriptsize
  \begin{verbatim}
Image Source: Input Floor Plan Image
Detection: door | Confidence: 0.82
Bounding box: [685, 1993, 773, 2084]
Center: (729, 2038)
Room A: HALL | Room B: SEJOUR
Wall side: South wall of HALL
Door type: hinged single door
Description: Standard interior door connecting
  HALL to SEJOUR, on the south wall of HALL
 ...
  \end{verbatim}
  }
  \end{tcolorbox}
  \caption{Components of the multimodal knowledge base: graph-based relational knowledge, vector-based semantic knowledge, and visual grounding context.}
  \label{fig:three-tier}
\end{figure}




%=====================================================================
\section{Experiments}\label{sec:experiments}

\subsection{Setup}

% \paragraph{Dataset.}
We ran the system on a MacBook Pro equipped with an Apple M1 Pro chip, featuring a 10-core CPU, 16-core GPU, and 16 GB of unified memory. Live video was streamed from a head-mounted camera directly to the local system, where the ArUco detection module processed each frame to identify fiducial markers~\cite{garrido2014automatic} distributed throughout the environment. Each marker encodes a unique node ID, anchoring the user's position within the knowledge graph and enabling accurate routing to destinations. User queries are transcribed using Whisper~\cite{whisper}, simplifying the query process. Navigation performance is quantified using Success Rate~(SR), a standard benchmark metric defined as:
\begin{equation}
    SR = \frac{N_{\text{success}}}{N_{\text{total}}}
\end{equation}
where $N_{\text{success}}$ is the number of completed navigations and $N_{\text{total}}$ 
is the total number of trials.

\begin{figure}[!t]
    \centering
    \includegraphics[width=\columnwidth]{FP-1.png}

  \caption{Bar plot comparing zero-shot navigation accuracy across short, medium, and long routes on CVC-FP for different models. }
  \label{fig:CVC-FP}
\end{figure}

\subsection{Evaluation on CVC-FP Benchmark}

We evaluate our proposed method using Claude~3.7~\cite{anthropic2024claude} Sonnet, GPT-4o, and LLaMA~3.2 Vision-Instruct~11B~\cite{llama3.2}, which we obtained from our prior study. A trial was considered successful when the system produced navigation instructions that accurately directed the user from the starting point to the destination without passing through structurally invalid transitions or entering inaccessible areas. As presented in Figure~\ref{fig:CVC-FP}, our system achieves the highest 
Success Rate~(SR) across all route types: 89.4\%, 83.2\%, and 71.5\% for short, medium, and long paths, respectively. These results surpass our prior baselines: 84.62\%, 69.23\%, and 53.85\% for Claude~3.7 Sonnet on short, medium, and long routes, respectively. LLaMA~3.2 records the lowest performance at 71.3\%, 61.4\%, and 42.6\% for short, medium, and long routes, respectively, on the CVC-FP dataset.




\subsection{Evaluation on MP Building}\label{sec:realworld}

To validate the system on real-world floor plans, a sighted evaluator physically walked 39 routes (13 short, 13 medium, 13 long) in the UMBC Math~\&~Psychology~(MP) building, following audio navigation instructions via screen reader. A route was marked successful if the evaluator reached the correct destination without deviating from the instructed path or encountering a safety hazard. Table~\ref{tab:MP1} reports results on MP-1, where the graph-augmented baselines (from our prior work~\cite{ayanzadeh2025floorplan2guide}) consistently outperform their graph-free counterparts. Our method achieves the best SR of 92.31\%, 76.92\%, and 61.54\% for short, medium, and long routes on MP-1, respectively. We further extend this evaluation to the larger and more complex MP-3 floor plan; results are reported in Table~\ref{tab:MP3_graph_ablation}. As shown in Table~\ref{tab:MP3_graph_ablation}, our system outperforms all baseline models on MP-3, achieving SR values of 76.92\%, 61.54\%, and 38.46\% for short, medium, and long routes, respectively. Claude~3.7 Sonnet reaches 61.54\% on short routes but falls to 23.08\% on long paths. GPT-4o achieves 61.54\% on short routes but degrades to 7.69\% on long routes. LLaMA~3.2 records the lowest performance, scoring 46.15\%, 15.38\%, and 7.69\% for short, medium, and long paths. Compared with MP-1, all models show decreased SR on MP-3 due to the larger number of rooms and increased layout complexity. Occasional OCR mislocalization degrades graph extraction quality and consequently navigation performance.



\begin{table}[htbp]
\centering
\caption{Success rate comparison on MP-1 in Zero-shot prompting,
with and without spatial knowledge graph augmentation.
Numbers in parentheses indicate successful trials out of 13 total
routes. Bold rows denote graph-augmented conditions.
Our method has no graph-free variant as the graph is a core
architectural component.}
\label{tab:MP1}
\footnotesize
\setlength{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.10}
\begin{tabular}{|l|c|c|c|c|}
\hline
\textbf{Model} & \textbf{Graph} & \textbf{Short (\%)} & \textbf{Medium (\%)} & \textbf{Long (\%)} \\
\hline
\multirow{2}{*}{Sonnet 3.7~\cite{ayanzadeh2025floorplan2guide}}
  & $\times$ & 76.92 (10) & 61.54 (8)  & 38.46 (5) \\
  & \checkmark & \textbf{84.62 (11)} & \textbf{69.23 (9)}  & \textbf{53.85 (7)} \\
\hline
\multirow{2}{*}{GPT-4o~\cite{ayanzadeh2025floorplan2guide}}
  & $\times$ & 61.54 (8)  & 46.15 (6)  & 30.77 (4) \\
  & \checkmark & \textbf{69.23 (9)}  & \textbf{61.54 (8)}  & \textbf{46.15 (6)} \\
\hline
\multirow{2}{*}{LLaMA 3.2~\cite{ayanzadeh2025floorplan2guide}}
  & $\times$ & 46.15 (6)  & 23.08 (3)  & 15.38 (2) \\
  & \checkmark & \textbf{53.85 (7)}  & \textbf{38.46 (5)}  & \textbf{30.77 (4)} \\
\hline
\textbf{Our Method} & \checkmark & \textbf{92.31 (12)} & \textbf{76.92 (10)} & \textbf{61.54 (8)} \\
\hline
\end{tabular}
\end{table}


% \begin{table}[htbp]
% \centering
% \caption{Navigation success rate for short, medium, and long routes on MP-3. Values in parentheses indicate successful trials out of 13.}

% \label{tab:mp2} 
% \footnotesize
% \setlength{\tabcolsep}{5pt}
% \renewcommand{\arraystretch}{1.10}
% \begin{tabular}{|l|c|c|c|}
% \hline
% \textbf{Model } & \textbf{Short (\%)} & \textbf{Medium (\%)} & \textbf{Long (\%)} \\
% \hline
% Claude 3.7  & 61.53 (8) & 46.15 (6) & 23.07 (3) \\
% GPT-4o     & 61.54 (8)  & 38.46 (5)  & 7.32 (1) \\
% LLaMA 3.2  & 46.15 (6)  & 15.38 (2)  & 7.32 (1) \\
% Our Method   & \textbf{76.92(10)}  & \textbf{61.54 (8)} & \textbf{38.46(5)} \\
% \hline
% \end{tabular}
% \end{table}

\begin{table}[htbp]
\centering
\caption{Success rate comparison on MP-3 in Zero-shot prompting,
with and without spatial knowledge graph augmentation.
Numbers in parentheses indicate successful trials out of 13 total
routes. Our approach has no graph-free variant as the graph is a
core architectural component.}
\label{tab:MP3_graph_ablation}
\footnotesize
\setlength{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.10}
\begin{tabular}{|l|c|c|c|c|}
\hline
\textbf{Model} & \textbf{Graph} & \textbf{Short (\%)} & \textbf{Medium (\%)} & \textbf{Long (\%)} \\
\hline
\multirow{2}{*}{Sonnet 3.7}
  & $\times$ & 46.15 (6)  & 46.15 (6)  & 15.38 (2) \\
  & \checkmark & \textbf{61.54 (8)}  & \textbf{46.15 (6)}  & \textbf{23.08 (3)} \\
\hline
\multirow{2}{*}{GPT-4o}
  & $\times$ & 38.46 (5)  & 23.08 (3)  & 7.69 (1)  \\
  & \checkmark & \textbf{61.54 (8)}  & \textbf{38.46 (5)}  & \textbf{7.69 (1)}  \\
\hline
\multirow{2}{*}{LLaMA 3.2}
  & $\times$ & 23.08 (3)  & 7.69 (1)   & 0.00 (0)  \\
  & \checkmark & \textbf{46.15 (6)}  & \textbf{15.38 (2)}  & \textbf{7.69 (1)}  \\
\hline
\textbf{Our Method} & \checkmark & \textbf{76.92 (10)} & \textbf{61.54 (8)} & \textbf{38.46 (5)} \\
\hline
\end{tabular}
\end{table}

\subsection{Limitations and Future Work}In this study, we evaluate the proposed model on a single architectural layout; to extend this approach to multi-floor buildings will require integrating logic for elevator and staircase connectivity. Real-time localization, which combines user position data with floor plan navigation, remains unaddressed. This system relies on a fiducial marker~\cite{garrido2014automatic} as a localization checkpoint to assist users during navigation and does not use devices which are equipped with sensors such as LiDAR or IMU, which could enhance positioning accuracy and reduce rerouting associated with the exclusive use of ArUco markers. A preliminary real-world evaluation of UMBC buildings (Section~\ref{sec:realworld}) demonstrates that the system generates physically valid and navigable routes; however, the pilot study included only a single sighted evaluator. To address these limitations, a user study with BLV participants is required to evaluate the system's accessibility and usability in dynamic indoor environments. Moreover, in this study, we focus on the SR as the main metric by considering the importance of having accessible and safe navigation to reach the destination; to extend our study, besides the user study, we will include the different metrics, such as hallucination rate, rerouting metrics, and other related metrics, to measure the reliability of the proposed system. All future human-subject studies will be conducted under approved Institutional Review Board (IRB) protocols to ensure informed consent and participant safety.



%=====================================================================
\section{Conclusion}\label{sec:conclusion}

We present a system that converts a single architectural floor plan image into accessible, safety-aware indoor navigation instructions. Our agentic pipeline features self-correcting mechanisms to ease the robust generation of spatial knowledge graphs from the floor plan. The multimodal knowledge base grounds navigation generation in graph-based, semantic, and visual contexts and is employed by a planner agent, which is evaluated by a safety evaluator that has feedback loops acting as a built-in. We achieve navigation success rates of 92.31\% and 76.92\% on short routes in MP-1 and MP-3, respectively, improving over the best single-call baselines of 84.62\% (MP-1) and 61.54\% (MP-3). On the CVC-FP benchmark, we obtained 89.4\%, 83.2\%, and 71.5\% for short, medium, and long routes, consistently outperforming all single-call LLM baselines. This study demonstrates that large language models, when integrated within an agentic framework, can serve as a powerful backbone for floor plan parsing and accessible path planning, providing a scalable solution toward infrastructure-free, safe, and scalable indoor navigation for the BLV community.

%=====================================================================

\paragraph{Ethical Considerations.}

LLMs were used only to improve language clarity and readability. All research design, experimentation, and analytical decisions were performed independently by the authors.

