1
0
mirror of https://github.com/mgerb/mywebsite synced 2026-01-11 10:22:53 +00:00

finished converting posts - need to work on resume

This commit is contained in:
2016-08-17 23:36:05 -05:00
parent 70807c7a45
commit f71ac40e32
7 changed files with 228 additions and 246 deletions

View File

@@ -1,55 +0,0 @@
<div class="blog-post">
<h2 id="title" class="blog-post-title">Temperature Sensor - Server Side</h2>
<p id="date" class="blog-post-meta">January 1, 2016 by Mitchell</p>
<p id="intro">The server side coding is a bit more complicated than programming the ESP8266 itself. I use a NoSQL database to store the information and some of the queries are complex and can be confusing to understand. I also make use of REST API's to send the data to the client side.
</p>
<hr>
<h3>Storing Information</h3>
<p>For this project I am using MongoDB to store all of my information because it goes well with my server, which is coded in Node.js, but any database could be used. I also wanted to learn about NoSQL databases because I already have experience with SQL databases. Information on coding the ESP8266 can be found <a href="/?post=12-18-2015.html">here</a> in a previous post.
</p>
<code>mitchellg.me/temperature?temperature=0&humidity=0&location=0&key=0</code>
<p>Above is the GET request that is sent to the server from the ESP8266. Data is tranferred through GET parameters. The key is an authentication code that I set to prevent unwanted HTTP requests. This is similar to an API key. Below is the code implemented to handle the HTTP request from the ESP8266.
</p>
<script src="https://gist.github.com/mgerb/4bedafa2bde264ee3135.js"></script>
<h3>Creating the REST API</h3>
<p>There are many ways that I could display the temperature information on a graph, but I was trying to come up with a quick and easy way that was also efficient. I also wanted reusability in case I wanted to add or change things down the road. I decided I was going to use <a href="http://www.chartjs.org/">Chart.js</a> because it is open source and free. These graphs are implemented on the front end using javascript. Because of this I needed to figure out a way to send the sensor data to the client side. I felt that the best way for me to do it would be creating a REST API. I plan on making another post in the future explaining all of my client side code along with how to use Chart.js.
</p>
<p>Now that I know how I want to display my information I just need to think of what information I want to display. I thought it would be cool to display a few different graphs. As of right now I have one graph that displays data by the year and one that displays by each month. The maximum and minimum temperature of each day are displayed as well as the average humidity for that day. The user can also select which year or which month to display and the graph will adjust accordingly.
</p>
<p>Now that I know what information I need I can start developing my REST API. I have a MongoDB collection called "temperature", which stores temperature, humidity, location, and time updated. Updated is a type of Date, which I will use in all of the queries to group by each day. To do the grouping in MongoDB I needed to use the <a href="https://docs.mongodb.org/v3.0/aggregation/">aggregation functionality</a> of MongoDB. Aggregation allows me to essentially perform queries on top of queries using the MongoDB "pipeline". This is similar to an SQL query when a selection is performed within a selection.
</p>
<script src="https://gist.github.com/mgerb/4879f4897f7e863e9003.js"></script>
<h3>Explanation of MongoDB Queries</h3>
<p>The first function takes in a location and year and returns maximum and minimum temperature readings as well as average humidity for each day in the selected year. The results are also returned sorted from newest to oldest.
</p>
<p>In the first part of the aggregation pipeline, which is $project, I am just selecting the temperature, humidity, year, month, and day. This part is important because it allows me to pull the year, month, and day from the date that is stored in the database. This way I do not have to store each of these entries separately in the database.
</p>
<p>This data is then passed to the next part of the aggregation pipeline. The $match pipeline stage selects out information from the collection which match with the selected year and location.
</p>
<p>The $group aggregation operator is the stage in which I actually group the data by the location, year, month, and day. For each group I also take the max and min temperatures along with the average humidity by using the correct accumulator operators. Now that I have the appropriate information I need, I just use the $sort pipeline operator to sort the data based on time updated.
</p>
<p>The function to display by month is the exact same, except I take in the month attribute and add it to the $match operator within the aggregation pipeline. Just like that the query is complete and all I need to do is send the response back to the client. Chart.js uses JSON format, which makes things extremely easy because MongoDB querie results are in JSON. I set the content type to JSON, and use JSON.stringify() to convert the JSON to a readable format for debugging purposes. An example API request can be tested out here.
</p>
<p><a href="/api/sensorbylocation/year?location=Winona%20Apartment&year=2016">/api/sensorbylocation/year?location=Winona%20Apartment&year=2016</a>
</p>
</div>

View File

@@ -1,125 +0,0 @@
<div class="blog-post">
<h2 id="title" class="blog-post-title">Hosting a Node.js Web Server on Digital Ocean</h2>
<p id="date" class="blog-post-meta">January 4, 2016 by Mitchell</p>
<p id="intro">A VPS or "Virtual Private Server" is a nice way to host a web server. Digital Ocean provides a great service based on my experiences. I recently rebuilt a new server to host this site and I will go over the entire setup process.
</p>
<hr>
<h3>Creating a Droplet</h3>
<p>Digital Ocean calls their VPS's "Droplets" and they are very simple to set up. Create an account on the site and attach a credit card. You are now ready to create a droplet. In my case I use an Ubuntu server but Digital Ocean offers a variety of linux operating systems. Select the size, location, and name and the server is up and running! You will be emailed with the login information.
</p>
<h3>Login with SSH</h3>
<p>Select your newly created droplet and navigate to console access. You will be prompted to change your password. Once the password is changed you can connect to the root user account via SSH.
</p>
<pre><code>
$ ssh root@&lt;ip address&gt;
</code></pre>
<p>If you are using a Windows machine you can download <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html">putty</a> to connect with SSH. If you choose not to use SSH you can just use the console access on the Digital Ocean website.
</p>
<h3>Firewall</h3>
<p>The very next thing I do is set up the firewall. I only want to open ports that need to be open. I use the built in firewall tool in Ubuntu called UFW.
</p>
<pre><code>
$ ufw enable
</code></pre>
<p>Now that UFW is enabled I can start opening the necessary ports. I must open 22 if I want to continue to SSH into the server.
</p>
<pre><code>
$ ufw allow 22/tcp
</code></pre>
<p>This will open TCP port 22 for SSH. I also open port 80 because that is the port in which the web server will run on.
</p>
<h3>Network Monitoring</h3>
<pre><code>
$ sudo apt-get install vnstat
</code></pre>
<p>VNStat is a nice tool that will allow me to monitor the bandwidth my server consumes both outbound and inbound. Simply use the command "vnstat" to check the bandwidth. Note that it takes awhile for vnstat to capture bandwidth data if you just upon first installing it.
</p>
<h3>Terminal Multitasking</h3>
<pre><code>
$ sudo apt-get install screen
</code></pre>
<p>Screen is a really nice tool too use when dealing with multiple terminal windows. Simply use the command <code>$ screen</code> to start screen. Press CTRL>A then C to open another screen window. Navigate between screens by pressing CTRL>A then N. To check if scren is attached or detached simply use the command <code>$ screen -ls</code>. Reattach to screen by using <code>$ screen -r</code> and detach with <code>$ screen -d</code>. If screen is open and already attached, but you wish to attach to it, simply use the command <code>$ screen -d -r</code>.
</p>
<h3>NPM</h3>
<p>Now that I have the basics set up on my server I can start installing the dependencies for my Node.js web server. NPM is a package manager that is needed for Node.js. It makes things very easy when used properly.
</p>
<pre><code>
$ sudo apt-get install npm
$ sudo npm update npm -g
</code></pre>
<p>The package manager on ubuntu does not have the most recent NPM packge as of right now but it is very easy to update with NPM itself. The second command listed above will download the newest version of NPM and update itself.
</p>
<h3>Node.js</h3>
<p>Now that NPM is installed I can use it to install Node.js.
</p>
<pre><code>
$ sudo npm cache clean -f
$ sudo npm install -g n
$ sudo n stable
$ sudo ln -sf /usr/local/n/versions/node/&lt;VERSION&gt;/bin/node /usr/bin/node
</code></pre>
<p>This will install the latest version of Node.js.
</p>
<h3>MongoDB</h3>
<p>Installing MongoDB used to be very simple, but now they do not include the init scripts with the newest version. Installation instructions can be found <a href="https://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/">here</a>.
</p>
<p>Once MongoDB is installed it can be run by using the command <code>$ mongod</code>, but I am going to show an easier way using the startup script. This way MongoDB will start when the VPS boots and it can also be monitored with the <code>$ service</code> linux command.
</p>
<pre><code>
$ cd /etc/init.d
$ sudo nano mongod
</code></pre>
<p>Once here copy the contents of <a href="https://github.com/mongodb/mongo/blob/master/debian/init.d">this file</a> into mongod. We now have the startup script, but we need to change some permissions first before it can run.
</p>
<pre><code>
$ sudo chmod 755 /etc/init.d/mongod
$ sudo chown root:root /etc/init.d/mongod
</code></pre>
<p>And to run MongoDB upon system startup.
</p>
<pre><code>
$ update-rc.d mongod defaults
</code></pre>
<h3>Restoring Database Files</h3>
<p>The web server is almost ready, but first I need to make sure all of my previous records are stored in the database. To do this I navigate to the folder which my records are stored in, which is contained in my git repository. I then use the command <code>$ mongorestore</code>, which loads everything from the dump into the freshly installed MongoDB. To backup MongoDB simply navigate to the desired folder to store the backup and issue the command <code>$ mongodump</code>.
</p>
</div>

View File

@@ -1,65 +0,0 @@
<div class="blog-post">
<h2 id="title" class="blog-post-title">Flashing the ESP8266 with the Arduino IDE</h2>
<p id="date" class="blog-post-meta">May 22, 2016 by Mitchell</p>
<p id="intro">Flashing the ESP8266 with the newest version of the ESP8266/Arduino can be tedious with the off brand FTDI programmers.
</p>
<hr>
<h3>Problems with knockoff FTDI programmers</h3>
<p>The new version of ESP8266/Arduino offers different flashing methods than the previous versions, which are the CK and NodeMCU methods. These new methods allow the flashing tool to automatically reset the ESP8266 into boot loader mode. This is convenient if you have a NodeMCU or the right FTDI programmer. I have a cheap knock off FTDI programmer in which the CK flashing method does not work with.
</p>
<p>The table below shows the correct connections for using the esptool-ck method of flashing. The FTDI programmer that I use does not have the "RTS" pin. Alternatively it has a "CTS" pin. I figured the FTDI programmers were the exact same, but after attempting to flash with this method I was convinced otherwise.
</p>
<p><a href="https://github.com/igrr/esptool-ck" target="_blank">The current version of ESP8266/Arduino uses the esptool-ck flashing method</a></p>
<div class="table-responsive">
<table class="table-bordered">
<thead>
<tr>
<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>none</td>
<td>No DTR/RTS manipulation</td>
</tr>
<tr>
<td>ck</td>
<td>RTS controls RESET or CH_PD, DTR controls GPIO0</td>
</tr>
<tr>
<td>wifio</td>
<td>TXD controls GPIO0 via PNP transistor and DTR controls RESET via a capacitor</td>
</tr>
<tr>
<td>nodemcu</td>
<td>GPIO0 and RESET controlled using two NPN transistors as in <a href="https://raw.githubusercontent.com/nodemcu/nodemcu-devkit/master/Documents/NODEMCU_DEVKIT_SCH.png">NodeMCU devkit</a>.</td>
</tr>
</tbody>
</table>
</div>
<h3>The Solution</h3>
<p>Unfortunately the current verion of ESP8266/Arduino does not offer alternative methods of flashing. After hours of frustration I finally figured out a solution to the problem.
</p>
<p>The methods to flash the ESP8266 within the Arduino IDE can be adjusted. I first thought I needed to download another method to flash with such as the <a href="https://github.com/themadinventor/esptool">esptool.py</a>, but I found a much easier solution. The "boards.txt" file can be edited, which allowed more customization to the flashing settings within the Arduino IDE. This is the path on my system.
</p>
<pre><code>/Users/$USER/Library/Arduino15/packages/esp8266/hardware/esp8266/2.1.0/boards.txt</code></pre>
<p>The reset method needs to be changed from "ck" to "none" in order for this to work properly.
</p>
<pre><code>generic.upload.resetmethod=ck</code></pre>
<p>to</p>
<pre><code>generic.upload.resetmethod=none</code></pre>
<p>Because we are not using the CK flashing method, we need to boot up the ESP8266 into boot loader mode manually.
</p>
<pre><code>
GPIO0 -> gnd
VCC -> 3.3v
CH_PD -> 3.3v
GND -> gnd
TX -> RX
RX -> TX
</code></pre>
<p>The ESP8266 must be powered by an external 3.3v power supply, because the FTDI programmer cannot supply enough current. Also keep in mind that the ground of the FTDI programmer MUST be connected to the same ground that the ESP8266 is connected to. If this is not done it will not flash.
</p>
<p>This should solve any problems flashing with a knockoff FTDI programmer.</p>
</div>

View File

@@ -4,7 +4,7 @@
"description": "My personal website built with React and Go",
"main": "index.js",
"scripts": {
"build": "webpack && babel-node metadata.js",
"build": "webpack & babel-node metadata.js",
"c9": "webpack-dev-server --port $PORT --host $IP --hot --content-base dist --history-api-fallback",
"dev": "webpack-dev-server --content-base dist --inline --hot --history-api-fallback",
"prod-linux": "export NODE_ENV=production && webpack -p && babel-node metadata.js",
@@ -39,6 +39,7 @@
"font-awesome": "^4.6.3",
"highlight.js": "^9.6.0",
"html-webpack-plugin": "^2.22.0",
"marked": "^0.3.6",
"node-sass": "^3.8.0",
"react": "^15.3.0",
"react-dom": "^15.3.0",

View File

@@ -0,0 +1,35 @@
# Temperature Sensor - Server Side
The server side coding is a bit more complicated than programming the ESP8266 itself. I use a NoSQL database to store the information and some of the queries are complex and can be confusing to understand. I also make use of REST API's to send the data to the client side.
* * *
## Storing Information
For this project I am using MongoDB to store all of my information because it goes well with my server, which is coded in Node.js, but any database could be used. I also wanted to learn about NoSQL databases because I already have experience with SQL databases. Information on coding the ESP8266 can be found [here](/?post=12-18-2015.html) in a previous post.
`mitchellg.me/temperature?temperature=0&humidity=0&location=0&key=0`
Above is the GET request that is sent to the server from the ESP8266\. Data is transferred through GET parameters. The key is an authentication code that I set to prevent unwanted HTTP requests. This is similar to an API key. Below is the code implemented to handle the HTTP request from the ESP8266.
## Creating the REST API
There are many ways that I could display the temperature information on a graph, but I was trying to come up with a quick and easy way that was also efficient. I also wanted reusability in case I wanted to add or change things down the road. I decided I was going to use [Chart.js](http://www.chartjs.org/) because it is open source and free. These graphs are implemented on the front end using javascript. Because of this I needed to figure out a way to send the sensor data to the client side. I felt that the best way for me to do it would be creating a REST API. I plan on making another post in the future explaining all of my client side code along with how to use Chart.js.
Now that I know how I want to display my information I just need to think of what information I want to display. I thought it would be cool to display a few different graphs. As of right now I have one graph that displays data by the year and one that displays by each month. The maximum and minimum temperature of each day are displayed as well as the average humidity for that day. The user can also select which year or which month to display and the graph will adjust accordingly.
Now that I know what information I need I can start developing my REST API. I have a MongoDB collection called "temperature", which stores temperature, humidity, location, and time updated. Updated is a type of Date, which I will use in all of the queries to group by each day. To do the grouping in MongoDB I needed to use the [aggregation functionality](https://docs.mongodb.org/v3.0/aggregation/) of MongoDB. Aggregation allows me to essentially perform queries on top of queries using the MongoDB "pipeline". This is similar to an SQL query when a selection is performed within a selection.
## Explanation of MongoDB Queries
The first function takes in a location and year and returns maximum and minimum temperature readings as well as average humidity for each day in the selected year. The results are also returned sorted from newest to oldest.
In the first part of the aggregation pipeline, which is $project, I am just selecting the temperature, humidity, year, month, and day. This part is important because it allows me to pull the year, month, and day from the date that is stored in the database. This way I do not have to store each of these entries separately in the database.
This data is then passed to the next part of the aggregation pipeline. The $match pipeline stage selects out information from the collection which match with the selected year and location.
The $group aggregation operator is the stage in which I actually group the data by the location, year, month, and day. For each group I also take the max and min temperatures along with the average humidity by using the correct accumulator operators. Now that I have the appropriate information I need, I just use the $sort pipeline operator to sort the data based on time updated.
The function to display by month is the exact same, except I take in the month attribute and add it to the $match operator within the aggregation pipeline. Just like that the query is complete and all I need to do is send the response back to the client. Chart.js uses JSON format, which makes things extremely easy because MongoDB query results are in JSON. I set the content type to JSON, and use JSON.stringify() to convert the JSON to a readable format for debugging purposes. An example API request can be tested out here.
[/api/sensorbylocation/year?location=Winona%20Apartment&year=2016](/api/sensorbylocation/year?location=Winona%20Apartment&year=2016)

View File

@@ -0,0 +1,93 @@
# Hosting a Node.js Web Server on Digital Ocean
A VPS or "Virtual Private Server" is a nice way to host a web server. Digital Ocean provides a great service based on my experiences. I recently rebuilt a new server to host this site and I will go over the entire setup process.
* * *
## Creating a Droplet
Digital Ocean calls their VPS's "Droplets" and they are very simple to set up. Create an account on the site and attach a credit card. You are now ready to create a droplet. In my case I use an Ubuntu server but Digital Ocean offers a variety of linux operating systems. Select the size, location, and name and the server is up and running! You will be emailed with the login information.
## Login with SSH
Select your newly created droplet and navigate to console access. You will be prompted to change your password. Once the password is changed you can connect to the root user account via SSH.
`$ ssh root@<ip address>`
If you are using a Windows machine you can download [putty](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html) to connect with SSH. If you choose not to use SSH you can just use the console access on the Digital Ocean website.
## Firewall
The very next thing I do is set up the firewall. I only want to open ports that need to be open. I use the built in firewall tool in Ubuntu called UFW.
`$ ufw enable`
Now that UFW is enabled I can start opening the necessary ports. I must open 22 if I want to continue to SSH into the server.
`$ ufw allow 22/tcp`
This will open TCP port 22 for SSH. I also open port 80 because that is the port in which the web server will run on.
## Network Monitoring
`$ sudo apt-get install vnstat`
VNStat is a nice tool that will allow me to monitor the bandwidth my server consumes both outbound and inbound. Simply use the command "vnstat" to check the bandwidth. Note that it takes awhile for vnstat to capture bandwidth data if you just upon first installing it.
## Terminal Multitasking
`$ sudo apt-get install screen`
Screen is a really nice tool too use when dealing with multiple terminal windows. Simply use the command `$ screen` to start screen. Press CTRL>A then C to open another screen window. Navigate between screens by pressing CTRL>A then N. To check if scren is attached or detached simply use the command `$ screen -ls`. Reattach to screen by using `$ screen -r` and detach with `$ screen -d`. If screen is open and already attached, but you wish to attach to it, simply use the command `$ screen -d -r`.
## NPM
Now that I have the basics set up on my server I can start installing the dependencies for my Node.js web server. NPM is a package manager that is needed for Node.js. It makes things very easy when used properly.
```bash
$ sudo apt-get install npm
$ sudo npm update npm -g
```
The package manager on ubuntu does not have the most recent NPM packge as of right now but it is very easy to update with NPM itself. The second command listed above will download the newest version of NPM and update itself.
## Node.js
Now that NPM is installed I can use it to install Node.js.
```bash
$ sudo npm cache clean -f
$ sudo npm install -g n
$ sudo n stable
$ sudo ln -sf /usr/local/n/versions/node/<VERSION>/bin/node /usr/bin/node
```
This will install the latest version of Node.js.
## MongoDB
Installing MongoDB used to be very simple, but now they do not include the init scripts with the newest version. Installation instructions can be found [here](https://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/).
Once MongoDB is installed it can be run by using the command `$ mongod`, but I am going to show an easier way using the startup script. This way MongoDB will start when the VPS boots and it can also be monitored with the `$ service` linux command.
```bash
$ cd /etc/init.d
$ sudo nano mongod
```
Once here copy the contents of [this file](https://github.com/mongodb/mongo/blob/master/debian/init.d) into mongod. We now have the startup script, but we need to change some permissions first before it can run.
```bash
$ sudo chmod 755 /etc/init.d/mongod
$ sudo chown root:root /etc/init.d/mongod
```
And to run MongoDB upon system startup.
```bash
$ update-rc.d mongod defaults
```
## Restoring Database Files
The web server is almost ready, but first I need to make sure all of my previous records are stored in the database. To do this I navigate to the folder which my records are stored in, which is contained in my git repository. I then use the command `$ mongorestore`, which loads everything from the dump into the freshly installed MongoDB. To backup MongoDB simply navigate to the desired folder to store the backup and issue the command `$ mongodump`.

View File

@@ -0,0 +1,98 @@
# Flashing the ESP8266 with the Arduino IDE
Flashing the ESP8266 with the newest version of the ESP8266/Arduino can be tedious with the off brand FTDI programmers.
* * *
## Problems with knockoff FTDI programmers
The new version of ESP8266/Arduino offers different flashing methods than the previous versions, which are the CK and NodeMCU methods. These new methods allow the flashing tool to automatically reset the ESP8266 into boot loader mode. This is convenient if you have a NodeMCU or the right FTDI programmer. I have a cheap knock off FTDI programmer in which the CK flashing method does not work with.
The table below shows the correct connections for using the esptool-ck method of flashing. The FTDI programmer that I use does not have the "RTS" pin. Alternatively it has a "CTS" pin. I figured the FTDI programmers were the exact same, but after attempting to flash with this method I was convinced otherwise.
[The current version of ESP8266/Arduino uses the esptool-ck flashing method](https://github.com/igrr/esptool-ck)
<table class="table-bordered">
<thead>
<tr>
<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>none</td>
<td>No DTR/RTS manipulation</td>
</tr>
<tr>
<td>ck</td>
<td>RTS controls RESET or CH_PD, DTR controls GPIO0</td>
</tr>
<tr>
<td>wifio</td>
<td>TXD controls GPIO0 via PNP transistor and DTR controls RESET via a capacitor</td>
</tr>
<tr>
<td>nodemcu</td>
<td>GPIO0 and RESET controlled using two NPN transistors as in [NodeMCU devkit](https://raw.githubusercontent.com/nodemcu/nodemcu-devkit/master/Documents/NODEMCU_DEVKIT_SCH.png).</td>
</tr>
</tbody>
</table>
</div>
## The Solution
Unfortunately the current version of ESP8266/Arduino does not offer alternative methods of flashing. After hours of frustration I finally figured out a solution to the problem.
The methods to flash the ESP8266 within the Arduino IDE can be adjusted. I first thought I needed to download another method to flash with such as the [esptool.py](https://github.com/themadinventor/esptool), but I found a much easier solution. The "boards.txt" file can be edited, which allowed more customization to the flashing settings within the Arduino IDE. This is the path on my system.
`/Users/$USER/Library/Arduino15/packages/esp8266/hardware/esp8266/2.1.0/boards.txt`
The reset method needs to be changed from "ck" to "none" in order for this to work properly.
`generic.upload.resetmethod=ck`
to
`generic.upload.resetmethod=none`
Because we are not using the CK flashing method, we need to boot up the ESP8266 into boot loader mode manually.
```bash
GPIO0 -> gnd
VCC -> 3.3v
CH_PD -> 3.3v
GND -> gnd
TX -> RX
RX -> TX
```
The ESP8266 must be powered by an external 3.3v power supply, because the FTDI programmer cannot supply enough current. Also keep in mind that the ground of the FTDI programmer MUST be connected to the same ground that the ESP8266 is connected to. If this is not done it will not flash.
This should solve any problems flashing with a knockoff FTDI programmer.