Turning the Internet into Data
A Catalog Web API
The CUNY+ Catalog
A rich vein of data.
Search Results
Holdings Records
MARC Records
010 ## $a ###89048230 020 ## $a 0316107514 : $c $12.95 020 ## $a 0316107506 (pbk.) : $c $5.95 ($6.95 Can.) 040 ## $a DLC $c DLC $d DLC 050 00 $a GV943.25 $b .B74 1990 100 1# $a Brenner, Richard J., $d 1941- 245 10 $a Make the team. $p Soccer : $b a heads up guide to super soccer! / $c Richard J. Brenner. 246 30 $a Heads up guide to super soccer 250 ## $a 1st ed. 260 ## $a Boston : $b Little, Brown, $c c1990. 300 ## $a 127 p. : $b ill. ; $c 19 cm. 650 #0 $a Soccer $v Juvenile literature. 650 #1 $a Soccer.
Problem
How could this data be accessed programatically?
{
"title": "Methods for the analysis of large data-sets",
"author": "Di Ciaccio, Agostino.",
"year": "2012",
"library": "Hunter Main",
"docNumber": "007268706",
}<CD>
<TITLE>Greatest Hits</TITLE>
<ARTIST>Dolly Parton</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>RCA</COMPANY>
<PRICE>9.90</PRICE>
<YEAR>1982</YEAR>
</CD>Web Scraping
The DOM Tree
<html>
<head></head>
<body>
<h1>Header text here</h1>
<div>
<p>Paragraph text here</p>
<h2>Subheader text here</h2>
</div>
<div>
<p>Another paragraph here</p>
</div>
</body>
</html>
Behind the Curtain
app.get('/marc', function (req, res){
var docNumber = req.query["docNumber"];
var verbose = req.query["verbose"];
var uriBase = 'http://apps.appl.cuny.edu:83/F/';
verbose == '1' ? format = '001' : format = '002'
var options = {
uri: uriBase + '?func=direct&doc_number=' + docNumber + '&format=' + format
};
request(options, function(error, response, body) {
// debugger;
console.log(options.uri);
if (error && response.statusCode !== 200) {
console.log(error);
}
jsdom.env({
html: body,
scripts: [
'https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js'
]
}, function (err, window) {
// load jquery
var $ = window.jQuery;
$('img[src]').each(function(i,el){
$(el).removeAttr('src');
});
var wholeMarc = {};
var resultsTable = $('table')[4]; //Grabbing the MARC Record
var rows = $(resultsTable).children('tr');
rows.each(function (i, item) {
var marcLabel = $(item).children('td')[0];
var marcValue = $(item).children('td')[1];
var name = $(marcLabel).text().trim();
var value = $(marcValue).text().trim();
wholeMarc[name] = value;
});
res.writeHead(200, {
'Content-Type': 'text/plain',
'Access-Control-Allow-Origin' : '*'
});
res.end(JSON.stringify(wholeMarc, null, 2));
});
});
});using Nodejs
Documentation
The API
Currently located at:
mighty-wildwood-7308.herokuapp.comTo do a search for 'global warming' in Hunter College:
/search?query=global+warming&queryType=All+Fields&school=HUNTERDeeper Data
Applications
↓A New Catalog
- Mobile
- Responsive
- Clear UX
Collection Development Scripting
Given a list of ISBNs, for each item get:
- Title
- Price
- Holdings at all CUNY Libraries for this and any other edition
Automate Shelf-reading
Fun
Exploration
Serendipity
A Data Experiment
THE END
BY Stephen Zweibel
Turning the Internet into Data
A Catalog Web API
by Stephen Zweibel