Jen LooperThis article, part of a book in progress, was inspired by my work leading the Museums and Libraries pilot program at Microsoft
Machine Learning for Artisans: Classifying Lacemaking Techniques
The use of advanced inference techniques when visiting a museum collection may not be obvious. Why would machine learning be useful for analyzing antique textiles? Well, for one thing, collections management can greatly benefit from 'smart' tagging of collections that are in the process of being archived. Digitizing museum collections, a major area of effort over the past decade or so, has concentrated on bringing collections online, often piece by piece.
This work is of course vital to producing data that can be used by machine learning to improve models. The work is self-sustaining: the more items are brought online and digitized for public consumption, the better the models that can be produced from the ever-growing group of items. And the better the models, the easier it will be to catalogue items more quickly. Archival work might eventually be considerably less manual if advanced machine learning techniques can be applied to categorize and tag them.
In this article, we will look at some tools that help classify items according to their image. In the process, we offer a blueprint for building a bespoke classification system for categorizing items by their image. We can test a specific use case for determining categories of lace by their photo. By building a web app with a custom machine learning model that can be used offline, we can create a useful tool for an archivist.
A Fashionable Decoration
The process of twisting thread or strings together to produce new fabrics in a decorative openwork has been a handicraft since the ancient Egyptians. Lacemaking evolved to become a highly sophisticated art form. From the Renaissance through the Edwardian Era, handmade lace has been the last word in luxury. Widespread throughout Europe for hundreds of years, lace evolved alongside changing fashions.
Sumptuary laws often attempted to regulate who could wear the costly material. Still, a 'Merveilleuse', 'dandy' or 'Macaroni' could ruin a fortune by splurging in expensive imported lace at the height of fashion. Louis XIV's astute finance minister, Colbert, realized how much money was spent on this irresistible finery. He helped launch the powerhouse of French lacemaking by importing lacemakers from Venice to teach new techniques to local lacemakers. Thus, according to his plan, money would be spent on domestic lace, reducing the desire for imported finery. From the 17th to 19th centuries, European countries vied with each other to create the most fashionable and beautiful patterns and styles.
A 19th century 'Duchesse' or 'Princess' style collar from the author's private collection from Flanders
An amusing poem, written by someone in the circle of Mme. de Sevigny, describes the despair of the anthropomorphized foreign laces when they were told they must return to their countries - for fear of being torn to shreds by angry lacemakers - due to Colbert's edict of 1660:
C’est aujourd’huy, noble assistance,
Qu’il faut abandonner la France,
Et nous en aller bien et beaux,
Pour n’estre pas mis en lambeaux.Today, dear friends,
We must abandon France
And depart post-haste
So as to not be torn to pieces.La Revolte des Passemens c. 1661
A collar of Alençon lace, made in France, from the author's private collection
Nowadays, much historical lace remains locked up in private collections or folded away in the textile departments of museums. Managing and enhancing such a collection involves a sophisticated level of connoisseurship to detect the differences between, for example, the basic categories of needle lace vs. bobbin lace. Differentiating between the subtle variations of various regional net grounds, for example, is crucial to understanding provenance. Alençon lace's hexagonal twisted bars differ from Valencienne's diamond mesh or réseau.
Creating a machine learning solution that will recognize categories of lace presents an interesting challenge that will allow the would-be collector to discover both the promise, and the challenges, inherent in creating a reliable image recognition model. Before starting with data collection, it is instructive to understand the underlying concepts of image recognition via machine learning.
Getting started with image recognition
Your model is only as good as the question you ask of it. This means that if you have only a few images on which to train, or large gaps in your dataset, the model will not be able to understand what it is tasked with classifying. Say you have a dataset of antique buttons, for example, and want to determine which of these are the 'calico' style button. If you do not have good examples to show the model, it will not be able to answer your question 'Is this a calico button?'
For this reason, for your first iterations of your model creation, ask a simple question that reflects the dataset you are able to provide. In my personal possession are samples of antique lace. They tend to date from the 19th century, so have a particular style and are in varied condition. I propose to use the limited model I am able to create to be able to visit a larger collection of better examples and help classify a few types of lace. Since I have samples of Honiton, Duchesse, Alençon, Point de Paris, Venetian, and Coraline lace, these are the six classes on which I will train the model. Then, later, when I visit a museum, I can gather more data via video to improve and expand on the model by gathering more images and retraining it. In the meantime, it is helpful to have a web app that can run on your phone to run the model - offline if needed - to check its accuracy against new lace images.
We thus have a plan in place: to train a model for use in a web app. This means that two assets need to be built: a model and its web app.
The basics of image recognition
Before embarking on a machine learning project, it is useful to understand some of the vocabulary involved. Similarly, it is instructive to learn about some of the architectural choices that need to be made. Each of these have tradeoffs.
TensorFlow - Developed by Google, TensorFlow is an entire platform for machine learning, comprised of an ecosystem of tooling that helps researchers, data scientists, and developers develop and deploy machine learning models. TensorFlow has a version of its APIs that can be used directly by JavaScript developers called TensorFlow.js. TensorFlow models are also exportable in a 'lite' format for use in mobile applications and on edge devices such as Raspberry Pis. "The name TensorFlow derives from the operations that such neural networks perform on multidimensional data arrays, which are referred to as tensors". TensorFlow is an excellent choice for the web developer who wants to learn about machine learning by building apps.
model - A model is the file that is produced when machine learning algorithms have iterated over data, looking for patterns. The TensorFlow documentation defines it as "a function with learnable parameters that maps an input to an output". A good model has been trained on good data and gives accurate outputs for inputs it has not yet 'seen'.
weights - a 'weight' decides how much influence an input will have on an output.
training - given a dataset, split into 'training' and 'test' sets, the training process involvees the attempt to predict an output, given an input. Initially, the training process outputs many mistakes. By learning from these mistakes, the training process improves and outputs become more accurate. The iterative process of giving a machine learning process more and better data and retraining a model generally creates an increasingly accurate model.
pre-trained vs. custom - while creating a completely new model based on a completely new set of data is possible, in general the vast amount of data needed to generate a reasonably accurate model requires more compute and more data than is generally available to the individual practitioner. For this reason, many machine learning models can be generated from pre-trained models. These new models build on the 'knowledge' acquired by prior training. This new training can be done by using the concept of transfer learning. Transfer learning allows the solutions gathered by training one dataset to be applied to a second. For image recognition, this is a particularly useful strategy, as a new dataset can be used to train a model already trained on similar data.
Tools of the trade
To build an image recognition model, there are many tools at your disposal. The entire model can be built by hand using Jupyter notebooks and Python scripts, with Cloud compute for large models that need extensive training. Alternately, for small proofs of concept and to test the waters with machine learning, you can try several excellent low-code tools new to the market. One such free tool is Lobe.
Lobe is a free application that you download to your local computer and upload images to it for inference. All training and image storage is handled locally, so it is a very cost-effective solution. When your model gets bigger, however, you might want to work with a cloud provider (Google, Microsoft, or AWS, for example) for data and model management. Managing a machine learning model is an iterative process whereby you gather images and train a model on them sequentially. Lobe makes this process seamless by retraining automatically each time a new image is added and tagged, or each time an image is tested. If the model guesses the image incorrectly, the user is prompted to retag it and the model retrains. For small datasets where you want to have full control over how the model is handled locally, Lobe is a great tool.
As always, finding images on which to train a model is a challenge. For bespoke, museum-style datasets of unusual things, the challenge is doubled. A few strategies to gather images for training exist:
1. Use a browser extension to scrape images from the web. "Download All Images" extension is very useful; make sure the images can be used for your purpose if there is a license.
2. Take a video and splitting it into separate images per frame. Use FFMPEG to split a video by downloading the free ffmpeg library and converting your videos.
- If you have .mov video (from an iPhone, for example), convert the files to .mp4 using your computer's command line tools such as Terminal. Type
cdto go to the place where your file is found, and then type:ffmpeg -i movie.mov -vcodec copy -acodec copy out.mp4to convert the .mov file to an .mp4. - Next, take the .mp4 and convert each frame to a numbered image file by typing
ffmpeg -i out.mp4 img_%04d.jpg. A series of numbered images will be generated from the movie file.
3. Use a tool like Lobe to convert video as you work with your collection. Lobe includes a video tool that allows the user to take short videos of an object; the video is then automatically converted to images. Make sure you have good lighting and a good webcam to extract quality images. This is a good option to quickly create a large number of images based on your collection.
Train and test your model
Once you have gathered the items on which you want to train your model, use Lobe to collect their images either via image upload or via the video tool. Classify them by selecting groups of images and giving them a label. These are your classes. The model will train incrementally as you add images. When you are ready to test it, find some images online of the class you want to test, and drop them progressively into the 'play' area of Lobe's interface. Improve the model by indicating whether Lobe's guess as to the image's class is correct or incorrect.
Once you are satisfied with its accuracy, export it as a TensorFlow.js model. You can choose to optimize it prior, if you need to boost its accuracy a little more.
The model is exported into a folder. Included is some sample code, which you can delete (the example folder). There are most likely many numbered groupx-shard...bin files: these are the model's weights.
The model itself is contained in the model.json file. If you open this file you can determine that it is a graph-style model generated by TensorFlow and converted to TensorFlow.js, a library that allows web applications to leverage TensorFlow's APIs.
Build a web app to host your model
Now that the model is built, tested, and downloaded, the next step is to build a web app to host it. While models can be large files that might make your web app sluggish to start if they are particularly large, the beauty of hosting your model in a web application is that you can use it offline in a museum or collection context to classify items. Your web app will be able to run on a mobile phone and you will be able to scan samples of lace to get an idea of its class (as long as it falls in one of the classes on which your model was trained).
To build modern JavaScript web sites, you need Node.js (a JavaScript runtime) and npm (the Node package manager) installed on your machine. Your apps will run using Node.js, and you will use npm to install third party libraries and packages specific to your site's needs. Follow these instructions to install these tools.
A clean way to build a web app is by using Vue.js, a lightweight JavaScript framework particularly well-suited to scaffolding web sites quickly. Follow these installation instructions to get the Vue.js CLI (Command Line Interface) running on your local computer. Once it is installed, create a web site called 'lacemaking': vue create lacemaking. A series of prompts will be generated in your command line tool; follow these recommendations to build a web site using default settings.
When the app creation is complete, type cd lacemaking and then npm run serve in your command line or Terminal to view your new web site. Your site will run on port 8080 and you can visit it at http://localhost:8080.
Visual Studio Code, a free software package for developers, is extremely useful for building sites such as this, so make sure you have it, or some other IDE (Integrated Development Environment) available on your computer.
Import the model files
Your web site will have only one page, with a button to upload an image and background processes to handle the classification routines. Open the code that your CLI scaffolded, using Visual Studio Code.
First, create a folder in the public folder called models, and in that folder create a folder called lace. Put all the files generated by Lobe in this folder; the important ones are all the shard file and model.json. All the files in public are not processed by webpack, the library that builds your app; you want the model to be served as it, not compressed and built in any way, so it needs to stay in the non-built area.
Before going further, you will need to edit the
model.jsonso that it will run properly in your web app using TensorFlow.js. Openmodel.jsonand search for the line"modelInitializer": {"versions": {}},and remove it. After this your model will work with the web app.
Next, take the signature.json file created by Lobe and move it to the src/assets folder. This file contains important information about the model and you will use it in your app for various processes. It will stay in the assets folder so as to be available to be imported and used directly within the app for the information it contains.
Prepare the app for TensorFlow with image uploading
Next, install TensorFlow.js using npm. Open a terminal within Visual Studio Code by selecting Terminal > New Terminal in the code editor. Also install a package that helps with file uploads, managing camera image uploading. In the terminal, type: npm install @tensorflow/tfjs and npm install vue-image-upload-resize.
Check your package.json file to make sure the libraries are installed in the dependencies area.
In src/main.js, the main file of the application, add the following lines on line 3:
import ImageUploader from 'vue-image-upload-resize';
Vue.use(ImageUploader);
This code initializes the uploader library. Next, you will start working in the components/HelloWorld.vue file, which is a Single File Component (SFC) containing a template for HTML code, a script block for JavaScript methods and data management, and a styles block for CSS styling.
Note, you could change the name of this SFC from HelloWorld.vue to 'Lace.vue' or any other name. If you do, you need to change the App.vue file accordingly.
Edit the script block to import all the packages this app needs by adding these lines directly underneath <script>:
import * as tf from "@tensorflow/tfjs";
import signature from "@/assets/signature.json";
const MODEL_URL = "/models/lace/model.json";
The app is now ready to use TensorFlow.js from the TensorFlow package, the data from the signature file, and the model, loaded into a constant variable for use.
Use TensorFlow.js within the app
Next, add a data object under the name line in <script>:
data() {
return {
prediction: "",
model: "",
preview: "",
hasImage: false,
alt: '"",
image: null,
outputKey: "Confidences",
classes: signature.classes.Label,
shape: signature.inputs.Image.shape.slice(1, 3),
inputName: signature.inputs.Image.name,
};
},
This important block of code contains the defaults of all the variables used by this file. It includes a placeholder for the predictions returned by the model, the model itself, and data returned by the image uploader. It also manages elements imported via the signature.json file, especially the array of classes (Honiton, Point de Venise, etc) that Lobe exported. It also imports the signature's image shape parameters.
After the final comma of the data object, add a methods object that encloses all the functions needed to perform inference against the model:
methods: {
setImage(output) {
this.prediction = "";
this.hasImage = true;
this.preview = output;
},
getImage() {
//step 1, get the image
const image = this.$refs.img1;
let imageTensor = tf.browser.fromPixels(image, 3);
console.log(imageTensor);
this.loadModel(imageTensor);
},
async loadModel(imageTensor) {
//step 2, load model, start inference
this.model = await tf.loadGraphModel(MODEL_URL);
this.predict(imageTensor);
},
dispose() {
if (this.model) {
this.model.dispose();
}
},
predict(image) {
if (this.model) {
const [imgHeight, imgWidth] = image.shape.slice(0, 2);
// convert image to 0-1
const normalizedImage = tf.div(image, tf.scalar(255));
let norm = normalizedImage.reshape([1, ...normalizedImage.shape]);
const reshapedImage = norm;
// center crop and resize
let top = 0;
let left = 0;
let bottom = 1;
let right = 1;
if (imgHeight != imgWidth) {
const size = Math.min(imgHeight, imgWidth);
left = (imgWidth - size) / 2 / imgWidth;
top = (imgHeight - size) / 2 / imgHeight;
right = (imgWidth + size) / 2 / imgWidth;
bottom = (imgHeight + size) / 2 / imgHeight;
}
const croppedImage = tf.image.cropAndResize(
reshapedImage,
[[top, left, bottom, right]],
[0],
[this.shape[0], this.shape[1]]
);
const results = this.model.execute(
{ [this.inputName]: croppedImage },
signature.outputs[this.outputKey].name
);
const resultsArray = results.dataSync();
this.showPrediction(resultsArray);
} else {
console.error("Model not loaded, please await this.load() first.");
}
},
showPrediction(classification) {
//step 3 - classify
let classes = Array.from(this.classes);
let predictions = Array.from(classification).map(function (p, i) {
return {
id: i,
probability: Math.floor(p * 100) + "%",
class: classes[i],
};
});
this.prediction = predictions;
//stop the model inference
this.dispose();
},
},
There are several steps here; walking through them, we note that:
1. The user clicks an button to upload an image, and setImage() is called. The output of that process sets the preview variable to be the image uploaded.
2. getImage() is called once the preview has been set to the image output. The image is drawn to screen using the reference this.$refs.img1 (which you will add to the template in the next step). The image is converted to a tensor, for reading by TensorFlow, using the tf.browser.fromPixels API. Then, the model is loaded and sent this tensor as a parameter.
3. Since the model is rather large, loadModel is called asynchronously. When it is loaded, the prediction process starts, using the image tensor.
4. The predict() method is called once the model is loaded, and the image is read and reshaped so that the model can read it in an understandable format. The image is centered, cropped and resized. Then, the reshaped image is fed to the model and a results array is generated from the model's analysis of the image.
5. Once a result is generated from the model, a predictions array is created with an analysis of the classes and their probability displayed and available to the front end.
6. Finally, the model is disposed, and memory freed up.
Build the front end
The front end of the application can be quickly built within the template tags. Overwrite everything in the current template tags, and replace it with the following markup:
<div>
<h1>Lace Inference</h1>
<img :alt="alt" :src="preview" ref="img1" @load="getImage" />
<div class="uploader">
<image-uploader
:preview="false"
:className="['fileinput', { 'fileinput--loaded': hasImage }]"
capture="environment"
:debug="1"
doNotResize="gif,jpg,jpeg,png"
:autoRotate="true"
outputFormat="string"
@input="setImage"
>
<label for="fileInput" slot="upload-label">
<figure>
<svg
xmlns="http://www.w3.org/2000/svg"
width="32"
height="32"
viewBox="0 0 32 32"
>
<path
class="path1"
d="M9.5 19c0 3.59 2.91 6.5 6.5 6.5s6.5-2.91 6.5-6.5-2.91-6.5-6.5-6.5-6.5 2.91-6.5 6.5zM30 8h-7c-0.5-2-1-4-3-4h-8c-2 0-2.5 2-3 4h-7c-1.1 0-2 0.9-2 2v18c0 1.1 0.9 2 2 2h28c1.1 0 2-0.9 2-2v-18c0-1.1-0.9-2-2-2zM16 27.875c-4.902 0-8.875-3.973-8.875-8.875s3.973-8.875 8.875-8.875c4.902 0 8.875 3.973 8.875 8.875s-3.973 8.875-8.875 8.875zM30 14h-4v-2h4v2z"
></path>
</svg>
</figure>
<span class="upload-caption">{{
hasImage ? "Replace" : "Click to upload"
}}</span>
</label>
</image-uploader>
</div>
<div>
<h2 v-if="prediction != ''">
<span v-for="p in prediction" :key="p.id">
{{ p.class }} {{ p.probability }}<br />
</span>
</h2>
<h2 v-else>
<span v-if="hasImage">Calculating...</span>
</h2>
</div>
</div>
This markup includes:
1. An image uploading tool available via the npm package installed earlier. This uploader calls the setImage() method to start the image processing routine.
2. An image placeholder where the uploaded image will be displayed for preview and analysis using the getImage() method. It is prevented from resizing the image, as that is handled in the reshaping routines.
3. An svg image of a camera that functions as a button and a caption that changes depending on whether an image has or has not yet been uploaded
4. An area below the image uploader to display predictions. If there are no predictions, a placeholder label is displayed.
Style the app
Finally, overwrite the entire style block to add a few basic styles to the app. This CSS code will create a stacked layout with an image, a button, and predictions.
<style>
#fileInput {
display: none;
}
h1,
h2 {
font-weight: normal;
}
ul {
list-style-type: none;
padding: 0;
}
li {
display: inline-block;
margin: 0 10px;
}
.uploader {
margin-top: 4rem;
margin-bottom: 4rem;
}
</style>
Run and Deploy the app
Run the app using npm run serve and see how it performs against various types of lace. If the model needs more data, or needs to improve, or if you want to add more classes, make the changes in Lobe. Then, re-import the output files into their proper places in your web app.
Your app is now ready to deploy to production so that you can use it 'in the wild', in a museum or a private collection. There are several options to deploy your app, many of which offer free web hosting. You might try Azure Static Web Apps or even GitHub pages, a solid choice and directly connected to your GitHub code. These solutions assume that you have committed your code to version control using GitHub, which you will need to do to deploy your app to the cloud.
Next Steps
You have successfully created a downloadable machine learning model built using Lobe, a quick way to use transfer learning locally to build an image recognizing tool. You also built an app to host the model and to use the camera to gather image data to identify various types of lace. What would be your next steps, to complete the typical machine learning circle of training, testing, analyzing, and retraining a model?
You might want to connect your Lobe models, as they are recreated from new data, to GitHub, so that you could schedule posting a fresh model on a schedule with new data. As your model grows and evolves, you could use your museum visits to gather more data and store it on your device, then feed it locally to Lobe and retrain a model. You could add more classes as you go, and your web app is flexible enough to handle their addition without needing to be edited. All you would need to do is find a way to refresh the model periodically, perhaps by means of a GitHub Action workflow that would be scheduled periodically.
These processes touch on the field of 'ML Ops' - the operational management of living machine learning models. As such they are outside the scope of this article, but by working with a small dataset and Lobe, you can see the promise of creating a model and helping it to evolve. In this way you broaden both its capabilities and your own knowledge about a collection.
Resources
Discover Lobe.ai!
History of Lace by Palliser, Bury, Mrs., 1805-1878; Dryden, Alice; Jourdain, Margaret
Lace and Lacemaking in the Time of Vermeer
La Révolte des Passemens, 1935, Published by the Needle and Bobbin Club of the Metropolitan Museum of Art.
