Drones above the Vineyards
So what are drones good for? Well a lot actually as it turns out. For example, I had the privilege of working with a top California Vineyard before harvest time, to investigate how Multirotor UAV’s could be used in vineyards to improve efficiency and identify crop issues. In the following I’ll highlight the workflow used, the results and some tips learnt the hard way.
Which UAV platform and Remote Sensing Equipment to use?
First off is what type of UAV to use, well it’s not as easy as picking up a drone off a store shelf for starters. To have useful photogrammetry results a number of key issues needs to be addressed:
1. Lift capacity and Endurance – Although this seems obvious, the UAV has to lift itself, batteries and camera/s into the air and fly the whole survey route. Working backwards, the UAV choice is therefore influenced by the payload or in this case the camera. I’ll explain more on this later, but for this test the payload weight was 8.15oz or 231g, or a point and shoot size camera. Add onto this a gimbal of approx. 100g; we needed to lift approx. 330g for approximately 12 minutes. Looking through specifications, the 3D Robotics Y6 was capable of this scenario, using a 4S 6000mAh battery http://store.3drobotics.com/products/3dr-rtf-y6-2014 Another option was the 3D Robotics X8, however we also wanted a copter that could fold down for transport, as such the Y6 with its foldable frame was selected over the X8.
2. Autonomous Flight with Flight Planning – When flying large areas of crops, flying manual and getting the correct overlap on images is near impossible. As such the UAV needs to be flown in an autonomous mode. This requires that the area to be mapped is stored in an electronic flight plan in the UAV, and the UAV flies from each assigned point or waypoint at a specific speed, altitude and orientation. This also allows the mapping mission to be flown, time and time again, days, weeks and months later. This is important, as it allows images from different dates to be compared side by side, allowing crop analysis over time. As we were flying 3D Robotics Y6 we had a choice of Pixhawk or APM autopilots. The UAV was initially a Y6A with AMP2.6, however for this mission the UAV was converted to a Y6B (mainly as this was better supported) but still with an APM2.6. In the future we will look at using the Pixhawk for the Y6. The APM2.6 has a long history in UAV autonomous flight, so this was the chosen platform.
3. Camera – The data is only as good as the images, as such the camera is critical. This data quality is a trade-off in a number of factors, weight, resolution, control, imager size, cost etc. We have a defined weight of approx. 200-300g as an acceptable payload weight. This places us in the point and shoot category. The camera must also have an NDVI capability. Also the camera needs to be setup with the correct parameters and also be triggered by the UAV, which the APM2.6 can do. This combination led us to use the Canon series of point and shoot cameras. Presently the SX260HS, S100 and S110 can be converted for NDVI and are used by companies such as Agribotix, Sensefly, Roboflight, Quest UAV etc. To simplify operation, the camera used was the SX260HS, as this has an on-board GPS, allowing for each image to be geo-tagged with GPS coordinates. This helps with the image processing later. The Canon range of cameras also are supported by an application called CHDK, which is placed on the cameras SD card. This supplies the camera with additional functionality, such as triggering from the UAV, interval timing shots, setting white balance etc. The camera is also 12.1Megapixel, for flying at a height of approx. 100 feet, with a ground resolution of approx. 1cm per pixel. More than enough for crop analysis, and flying up to 400 feet still gives excellent imagery for analysis. Finally as previously mentioned, the cameras need to be NDVI capable. This was achieved using an Event 38 NGB, near infrared, green, blue filter with the red spectrum notched out.
A tutorial on how to convert the camera is defined here, the process is relatively simple.
And the results of the conversion can be seen in this blog:
Other camera filters exist from companies such as Max-Max:
So the remote sensor is a Canon SX260HS with GPS, fitted with an Event 38 NGB NDVI filter, with CHDK software mounted on the camera SD card.
4. Camera Gimbal – Flight time is a trade-off of thrust versus weight, as such the lightest simplest quality gimbal was researched. Gimbal categories can be split into 3 main areas, simple servo gimbals, high quality servo gimbals sometimes with gearing, and brushless gimbals. The purpose of the gimbal is to allow the camera to take high quality images of the crop. To do this a number of issues must be addressed. Firstly the gimbal needs to keep the camera pointing straight down. This keeps the overlap on the images which I’ll explain later, even when the UAV is tilted when flying forwards or into crosswinds. This also stops blurring, as the camera is stabile in pitch and roll, thus not been moved around by the UAV movements. An important factor in this is the gimbal movement should be smooth. Secondly, the camera needs to avoid any vibration that could blur the images; therefore the camera needs to be isolated from vibrations of the Multirotor. This is normally achieved using rubber isolation grommets between the camera gimbal and the airframe. Thirdly, it should be light and simple, the more complex it is the more chance it will go wrong in the field. Finally it should be cost effective.
Based on these criteria, we need a smooth movement gimbal in pitch and roll, good vibration isolation, simple and light, and approx. $300. Simple servo gimbals although simple and light, can have sloppy or sharp movement, whilst brushless gimbals are very smooth as they are required for video work, tend to be more complex and heavier. As such a high quality servo gimbal was chosen, the GUAI Crane II which cost $279. It does not have any associated electronics as per brushless gimbals, instead using the UAV flight controller for gimbal control. An advantage of this gimbal is that it also allows the gimbal to be removed easily from the isolation damper for packing/traveling.
5. Camera setup – Again, the final analysis is only as good as the data used, which means you need good quality images. For UAV aerial images, there are a number of trade-offs, such as ISO settings, the aperture, auto-focus, shutter speed, white balance, image stabilization, image capture time etc. The main aim is to get a sharp image with the least amount of noise. Also the image quality is affected by the light quality, with results changing between a sunny day and a cloudy day for example (see Agribotix for further analysis on this.) Normally the following setting work and were used, white-balance sunny day, zoom set to wide angle to maximize image view, auto-focus off to speed time between images and focus set to infinity, aperture set to automatic, image stabilization off, shutter speed set to a medium such as 1/800, ISO set as low as possible to avoid noise.
Summary of Setup
OK, so we have the UAV the 3D Robotics Y6, the 3D Robotics APM2.6 autonomous autopilot, a Canon SX260HS with GPS, fitted with an Event 38 NGB filter, mounted in a GAUI Crane 2 gimbal.
Plan the Mission
The flight planning software for the 3D Robotics Y6 is called Mission Planner (http://copter.ardupilot.com/wiki/mission-planning-and-analysis/) and can be used to devise flight plans, configure the UAV, plus monitor the UAV in flight using a telemetry link. One useful point when planning a mission is to use a site survey if nearby, to access the safety and understand the terrain or any obstacles or special circumstances that need to be taken into account. Once this is done, Mission Planner can then be used to draw the survey map.
This survey grid is then converted to waypoints with flight altitudes, and uploaded into the UAV.
On the Day
Meeting the vineyard owner, a survey site was identified of approximately 7 acres, which kept the UAV away from trees, power lines, workers and an on-site event which was been setup. The site was focused on the middle of the vineyard with some elevation change involved. This is very common situation in the Santa Cruz area, where the vineyards propagate through the Santa Cruz Mountain region. Normally this is also associated with the vineyard been surrounded by tall trees such as Redwoods, which in turn leads to a large bird population. The upshot of this is the vineyard headache of birds been pests, which leads to most Santa Cruz vineyards using netting to protect their crop. However on this day only some of the crop was netted, so the majority of work was over the un-netted area.
Here is the Y6 ready to fly:
Josh Metz, UAV Observer and Vineyard GIS Specialist @Geovine :
As mentioned before the site had been pre-planned via Google Earth and a number of possible missions planned and stored. Therefore all that was required was to upload the correct mission the Y6. Firstly the NDVI NGB camera was loaded, turned on and allowed to get GPS. The mission was then started a survey grid flown with no issues, except one.
When the mission was preplanned, the landing site was in the center of the vineyard, however the take-off and landing site was moved to the top of a hill to get better observability of the UAV during the flight. The exception was that the take-off site was changed, but the landing site did not reset correctly. The result was at the end of the mission, the UAV attempted to land at the other end of the vineyard in the old landing spot. This was easily overcome by going to manual and flying it back and landing by hand. However the moral is, when you program a mission to a UAV, always read it back to make sure all changes are correct. Also always pay attention, have an observer in my case Josh Metz, Vineyard GIS Specialist @Geovine , and always be ready to take control back.
After the NDVI RGB camera flight, the camera was swapped out for a normal RGB camera, and the mission flown again. Again, this is the advantage of autonomous flight, both NGB and RGB doing two flights but over the same flight path.
A number of software suites exist to process images and create NDVI information, two examples are AgiSoft (http://agisoft.ru/ ) and Pix4D ( http://pix4d.com/ ). Another interesting choice is from Agribotix which is a Cloud Based NDVI service for post processing UAV images http://agribotix.com/
This survey was based on Pix4D who kindly gave us a Demo License for this investigation. The purpose of this software is numerous. Firstly it corrects for camera issues, as such the camera model used is added as input data, the separate images are uploaded and then the software joins all the separate images together in a point cloud. From this a single large image is generated and then numerous other outputs such elevation models, 3D models, plus NDVI data plots as outputs. The output formats are numerous, with Geo-Tiff been a primary output. To help the software align the images, ground control points can also be added, which gives known reference point for the software to stitch the images together.
To get a complete stitched image, all the images must overlap. The required overlap is normally 60% to 80% to allow the software to stitch properly. As such you need lots of images, and the lower you fly, the more images you need. The downside of this is that you must process more images, which takes more time. This process requires a fast computer using lots of memory, such as an Intel i7 running 32 or 64GB of memory. So two lessons fly as high as you can, but no higher than 400’, and have a very fast computer.
Pix4D did an excellent job of stitching the NGB NDVI images together, but issues did occur with the RGB images, although this was not a software issue. The GPS on the Canon SX260HS RGB camera had not geo-tagged the images correctly. However just using the Pix4D ground control points, the software was still able to stitch the RGB images together.
NGB Point Cloud showing the UAV position when the image was taken.
NGB Mosaic Image, showing the separate images stitched together.
RGB Point Cloud, where the images were stitched using just ground control points with no GPS data.
RGB Mosaic with all the stitched images.
The NDVI image was generated using the NGB bands processed by the Pix4D software. A quick explanation of the image brings to lights some details with NDVI imagery in vineyards. Firstly the dark blue is actually due to shadows on the ground between the vineyard rows. This was because the survey was flown in the morning around 10:30am, rather than noon with the sun directly overhead. The green indicates the ground. The red indicates the separate vines.
One of the main items to notice is where you have good vine virility and growth, you see red vines and the blue shadows. Where growth is low the vines in red are less obvious and the shadows (blue) less strong, and the ground (green) more merged together. Using this information it can be see that certain areas show lower growth and yields than other areas.
Correlating to Ground Data
After the data was processed, we went and talked with the vineyard owner, and compared ground data with our results. It was obvious from the discussions that the ground data and the UAV NDVI and RGB images both highlighted low yield areas, which were known to be lower than the rest of the vineyard due to soil type, irrigation etc. As such the UAV images with Pix4D processing were shown to have been able to correlate well to ground data.
It also became clear that the owner knew his vineyard very well, as it was approx. 17 acres, so he and his staff could walk the property and identify issues on the ground. As such UAV imagery only becomes effective as a business model when the property cannot be efficiently walked. At this point the UAV is indispensable in its ability to capture large areas and process data.
One advantage of UAV imagery that needs further investigation though, is that RGB images do not show yield issues that easily when the vines are netted, however preliminary analysis shows NDVI NGB images can show yield issues even when the vines are netted.
1. Keep it simple
2. If it can go wrong it will go wrong
3. Preplan your mission, do a site visit or use Google Earth for site info
4. The higher you fly, the less images you need which means less processing time
5. The higher you fly, the larger the area you can map
6. Always check your images when in the field
7. Fly at noon to limit shadows from the vines causing NDVI image
8. Use an observer
9. Crop analysis is 20% flying and 80% data processing
10. Image processing takes lots of computing power, get a fast processor with lots of memory
11. High quality images equates to high quality crop analysis, poor images mean poor data
12. Aerial images and analysis needs to be correlated with ground data to be effective
13. Normal photographs and video in RGB is almost as invaluable as NGB to the vineyard owner
14. Drone NDVI mapping becomes effective with vineyards greater than 50 acres
So we have shown how a UAV such as a 3D Robotics Y6, mounted with a simple Canon point and shoot camera modified with a NDVI filter, using powerful software such as Pix4D, can generate useful crop analysis for vineyards. We’ve pointed out lessons learned and are now ready to keep on helping the Santa Cruz Mountain Wineries stay the best in the World.