DETR Failure

I'm working on a project to bring advanced document intelligence to the construction industry.

After building an initial prototype I uncovered several sub-problems to dig into. One is how to cross-link references in construction drawings (think hyperlinking across pages). No doubt a capability that exists in the industry standard Procore, but I'm not using Procore.

The Goal

Train a custom object detection model using the DETR architecture to detect reference symbols in construction drawings.

Example reference symbol in a construction drawing


I read that DETR is elegant because it largely uses a transformer to solve object detection. That's great, transformers are being used for seemingly everything else and maybe this will help me understand them better.

There's a blog post by Facebook AI Research that explains the simplified architecture and I was excited to try it out.

I found this Hugging Face tutorial on how to fine-tune a DETR model which felt very approachable. Just bring your COCO dataset and go to town!

The Start

First I needed to create a COCO dataset. I have a few IFC (Issued for construction) drawings that I could use, but I needed to annotate them.

There are plenty of existing tools to annotate a dataset, but I had already started building a workbench tool to help with document processing and content extraction. The functionality to annotate images and export the annotations in COCO format would be a fun addition.

Fieldworks workbench tool

After a few days of work I had a tool that could load an image, draw bounding boxes, and export the annotations in COCO format. I was ready to start annotating. Just a couple of hours of manual labor and I had created my first dataset.

The Failure

I was ready to train the model. I followed the Hugging Face tutorial, modifying it to work with my local data. Finally, the big moment: I started the training process.

Since my dataset was miniscule the training went quick. After 3 minutes, out popped a model that I could use to make predictions. I loaded up a previously unseen drawing from my test set and ran it through the model.

The model predicted nothing. Not a single bounding box.

I tried a few more drawings. Nothing.

Action Items

I need to investigate further why this experiment failed.

Now that I have a COCO dataset and a tool to annotate more images, I can try out different architectures and see if I can get a model to work.

Stay tuned!

© Mike Surowiec