ANOTHER SMB vulnerability? Not for ONTAP!

Well well well.   Imagine that.  Microsoft is announcing another vulnerability in SMB.

For those of you that missed it:
CVE-2020-0796 is a unique ID assigned to a Microsoft-specific vulnerability in their SMB v3.1.1 compression code. https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0796

The question always arises: "Is ONTAP vulnerable?" 

Well....  what we do is take the vulnerability and assess it against ONTAP.  We make no assumptions and ensure that ONTAP is either not vulnerable or we log the vulnerability and fix it ASAP.  
Why is ONTAP not automatically vulnerable like Windows?  Doesn't it run SMB?   Well yes it does, but the code that runs ONTAP's SMB stack is proprietary NetApp code. It is a completely NetApp written stack.  ONTAP does not run or share any Microsoft SMB code.  If there are vulnerabilities in the Microsoft code, they are not necessarily in ONTAP code.  
There is, however, a possibility that the vulnerability exists in the protocol standard.  Any time there is a vulnerability in the implementation of a protocol it is possible that NetApp could make the same errors in the protocol implementation leading to a similar vulnerability. 

The bottom line:
For this particular vulnerability, it is in the SMB 3.1.1 compression feature implementation in Windows.  NetApp has determined that ONTAP is not vulnerable to this.  ONTAP does not support the SMB 3.1.1 compression feature and therefore is not vulnerable.

NOTE:   There is no need to post "not vulnerable" responses.  There will not be an official report that says ONTAP is not vulnerable.  

NFS and deny permissions

There's always "bonus features" that get stuck into products that may or may not be useful.   But the hard part of all this is when you start using that "bonus feature", then your locked in and... BANG!  They gotcha!  Then when you want to migrate off....   BOOM! You can't. 
Is it really locking you in?  Maybe... maybe not.  There are certain features that are life savers and you need to make a decision on whether or not to really abandon that product and go with another that doesn't have that feature.   For other features, you just need a way to do the same ting in another way.
Once of these features is a "deny permissions" setting on another NAS vendor's system.  This in essence denies anyone from being able to change the permissions of a file via NFS.  This means no chmod's allowed.  Is that a good thing?  Well maybe.  It may allow the permissions to be set initially on a file but all subsequent changes are denied.  This is great, but there's other mechanisms in the world to provide the same or similar behavior.  And why do you really need this?  For one, some unified permissions schemas are so insecure, this is a way to be "secure".


Deny Permissions on NFS the easy way.  With NTFS.
Lets look at ONTAP and the "deny permissions" flag.   How can you "deny permissions" for chmod in ONTAP?   Well, the easiest way is to create an NTFS volume (or change the security style of an existing volume).   If an NFS client tries to perform "chmod" operations to the files or directories in that volume, the operation will be denied.  This can cause problems with some Linux applications that can't have chmod return errors; like vi.  For this, ONTAP has a setting called "ignore security operations on NTFS volumes" (-ntfs-unix-security-ops ) .  When this is set, ONTAP will return a success to the chmod, which makes the application happy, but in reality, it just discards the chmod op.  This way, the only way to change the permissions is via Windows or SMB.  Now you can use Windows ACLs to further limit permissions and access.  Why is this good?  Well, you can set the initial permissions at the volume level and then be sure that all files and folders inherit the exact same permissions without fail.  If you don't allow SMB access, then noone will be able to change it.

In action?   Why sure.  Here we go.  (Assuming SVMs, CIFS server, protocols, and other Vserver is already done)
1.  Create an NTFS volume
cluster::> volume create -vserver svm0 -volume no_chmods -aggregate aggr1_node1 -size 100G -security-style ntfs -junction-path /no_chmods
2.  Change the setting to ignore the NFS chmods

cluster::*> vserver nfs modify -vserver svm0 -ntfs-unix-security-ops ignore

Then what happens?  Well you need to create the export policy for the volume, but then you just mount the volume and viola.....  there's no way to chmod.  You can't chown either from an NFS client.  You can only change the ACL from an SMB client.

So here I have the volume mounted, touched a file and then chmod'd it.  As you can see, there was no error to the chmod, just a return, but the chmod did nothing.  This is because when ONTAP is presenting an NTFS style volume to NFS clients, NFS security operations cannot  modify the file permissions.  

Bonus Material:   The NTFS permissions can even be locked down beyond what's listed via SLAG (Storage Level Access Guard) in ONTAP.  SLAG allows you to put another set of ACLs on the volume and further deny (SLAG can only limit and not allow more permission) access to the files.

Deny permissions on NFS with NFSv4 ACLs on UNIX style volumes.

You don't want to use NTFS style volumes?  There definitely is another way of limiting access to chmod, and that's via NFSv4 ACLs.  Those ACLs work similarly to NTFS ACLs and can deny rights to change permissions and ownership if necessary.  If you are not familiar with NFSv4 ACLs, go visit my favorite resource for them at die.net (https://linux.die.net/man/5/nfs4_acl).  There's some good resources I've relied on forever.  Here's the catch....  you can't limit the chmod for the file owner.  Bummer.  

How do you do this one?  Let's see.....

1.  Allow NFSv4 on the NFS server and allow the ACLs.  We're assuming you already have an SVM, an export policy and a volume exported.

cluster::> vserver nfs modify -vserver svm0 -v4.0 enabled -v4.0-acl enabled -v4.1 enabled -v4.1-acl enabled

2.  Enable the setting to force the ACL over mode bits to be applied on new files & folders

cluster::*> vserver nfs modify -vserver svm0 -v4-inherited-acl-preserve enabled

3.  Enable the setting to not clobber the ACLs when an actual chmod is run by someone who is allowed

cluster::*> vserver nfs modify -vserver svm01 -v4-acl-preserve enabled

Then you can mount the volume either NFSv3 or NFSv4 and the same ACL is applied to both protocols.  First you do have to set the ACL properly for this to work.  Let's see how this is done.

First set the proper permissions.  What you don't want in there is "write attributes" (capital T), write named attributes (capital N..  and it's optional) and write ACL (capital C)

# nfs4_setacl -a A::testerdlu0002@ntap.local:rwaxtncy testfile

This will allow the user to perform any operation except modify the ACL, owner and mode bits.

So now, when the user tries to chmod the file... viola.. permission denied!

Can you also do it with SLAG and UNIX style volumes??

Well sure you can.  You have to be a little careful with SLAG though.  There are a few caveats.  First, you have to have a CIFS server created on the SVM.  Easy enough.  Second, you MUST have Unix->Windows usermapping working perfectly.  Without it, SLAG won't be applied properly since it used the Windows user creds to apply it.  Third, it also looks to have an implicit default DENY rule at the end of it.  No bueno if you want to ensure the same access.

Here's the policy I was using that had some success.  Notice the 2 deny rules at the top.  The access rights hex value (c0110) translates to WriteEA,Write Attributes, Write DAC and Write Owner in the NTFS permission world.  This basically denies everyone the right to chmod.

cluster::*> file-directory ntfs dacl show -vserver svm0

  (vserver security file-directory ntfs dacl show)

Vserver: svm0

  NTFS Security Descriptor Name: deny_write_attr_everyone

    Account Name     Access   Access             Apply To

                     Type     Rights

    --------------   -------  -------            -----------

    BUILTIN\Users    deny     c0110             this-folder, sub-folders, files

    CREATOR OWNER    deny     c0110             this-folder, sub-folders, files

    BUILTIN\Administrators

                     allow    full-control      this-folder, sub-folders, files

    BUILTIN\Users    allow    full-control      this-folder, sub-folders, files

    CREATOR OWNER    allow    full-control      this-folder, sub-folders, files

    NT AUTHORITY\SYSTEM

                     allow    full-control      this-folder, sub-folders, files

6 entries were displayed.

The bottom line?   Well... to be honest, the only reason that some NAS appliances have this "deny chmod" feature is in order to keep files secure and to enforce a consistent set of permissions across a volume.  Well...  if you really want to do that, let's use NTFS volume security.  That's the easiest.  Then you can manage permissions once when the volume is created (apply inheritable permissions at the volume root) and then its applied everywhere.  If you REALLY Need it in a UNIX style volume.....  You can use NFSv4 ACLs to deny write attributes.  If you ABSOLUTELY need it to apply to the owner too, look into using SLAG.

Microsoft, ONTAP, and the LDAP Channel Binding Apocalypse

Howdy Folks!  I'm sure that everyone has heard and is panicking about the patch that Microsoft announced around LDAP communication and channel binding.  For those of you that are living under a rock with an early 2000's flip phone, here's the link: https://portal.msrc.microsoft.com/en-us/security-guidance/advisory/ADV190023

Well, let me be your guide around what this means when it comes to ONTAP.  Buckle up, and here we go.
                              

FIRST OFF:  LDAP CHANNEL BINDING TOKENS
The MS update (when it comes out.  as of this writing, it is now pushed to 2H CY2020) will require channel binding tokens(CBT) only if the client supports it.  That is the default registry setting of 1 that the patch will set.  So, when you install the patch and do nothing else, there will be no effect on ONTAP since ONTAP does not support CBT.   At some point, Microsoft may change the default setting to 2, or your AD team or security team may require a setting of 2, but until ONTAP supports CBTs, a setting of 2 is not compatible with ONTAP.   The default setting of 1 or a setting of 0 (never required) is compatible with ONTAP.
NOTE:  ONTAP above includes all releases of Cluster Mode and 7-Mode ONTAP and Data ONTAP.

NEXT:  THE REAL PROBLEM.....   LDAP SIGNING REQUIRED
The MS update will change the behavior of the settings for LDAP signing required.  This means that if you have never touched the setting, it will now behave as if LDAP signing is required in your environment.  Here's a little cheat matrix to help you.



If you explicitly set the LDAP Signing Requirement to "OFF" in your environment, you will have no impact after the patch.  Nothing will change.  The patch will not modify the setting.  Remember though, this will need to have been EXPLICITLY SET TO OFF either in a GPO or registry setting on the DCs.

If you are currently using "Signing required" in your environment (setting of 2), then you have already done everything in your environment necessary to support it.  The patch will not modify, enhance or change that setting.  You will have no impact after the patch.  Remember, you will have had to set this EXPLICITLY in a GPO or registry setting on the DCs.

For everything else, you need to do some changes.

I have an SVM with a CIFS server
If you are already using StartTLS, LDAPS, or signing/sealing for the CIFS security, you do not have to do anything.  Any of those settings will meet the requirements of the patch.  Here's an example SVM with none of the 3 set properly:

If you are like the above and have none of the possible settings set, the easiest setting to modify is the signing & sealing (client session security) option. This is because StartTLS and LDAPS both require certificates and if you don't have a certificate infrastructure in your environment, well...  its complicated.  In order to set signing use the following command:
 vserver cifs security modify -vserver svm0 -session-security-for-ad-ldap sign
Replace "svm0" with your vserver name.  That's it for the CIFS server.

This will only work for ONTAP 9 and higher.  For Data ONTAP 7-mode, the connection automatically upgrades to sign when it encounters an AD where signing is required.  There is nothing additional needed to do for 7-mode.  For ONTAP cluster-mode versions older than 9, you'll need to set up StartTLS.  There's plenty of documentation and resources on how to set up StartTLS for the CIFS server.  I'm not going to go into it here.

I have and SVM with a UNIX LDAP client
Here's where it gets a little complicated due to the amount of permutations of the LDAP config there are.  Now, what I can say is that the same rules apply for UNIX LDAP connections as the CIFS server security settings.  If you are already using StartTLS, LDAPS or signing/sealing for the configuration, then you are in the clear and you are meeting the requirements of the patch.  Here's a screenshot of a client config with none of the 3 set properly"

NOTE:   As of ONTAP 9.5, if you use "636"  for the LDAP Server Port, LDAPS will be used automatically

Now, I am not going to go through all the iterations of possible configurations and what their outcomes are, but I will point you to the best practice and what is the easiest; use the AD-Domain setting, use the bind-as-cifs-server setting, and use the signing & sealing (Client Session Security) option.  Using the AD-Domain setting will automatically discover AD DCs and attach to the closest one using sites and services.  Using the bind-as-cifs-server setting will guarantee a sasl bind and enable the ability to sign or seal.  Then using at least signing for that configuration will use signing for the LDAP connection similar to the CIFS server operation I explained above and satisfy the requirement.

Want to read the official NetApp KB article?  Click on this:  https://kb.netapp.com/app/answers/answer_view/a_id/1103575

**Whew**  Now that we've muddled though alot of the muck on this, I hope you followed me through the configurations and now you know what you need to do to stay operational once the patch hits and gets applied to your DCs.  Apocalypse.....  carry on!

To homedir or not to homedir.....

... that is the question.   Here's the answer:  WHY NOT!

What the heck am I talking about?  Well, its the lesser-used NetApp ONTAP homedir feature.  What the heck does it do?   Make home directory provisioning, configuration and management so much simpler!

So, that's a pretty bold blanket statement, but the benefits of using the homedir feature in ONTAP is just so overwhelming, there's really no downside (besides the initial configuration and user provisioning process modification) to enabling and using it.  In essence, it's really a best practice when using ONTAP for home directory serving.

What is this homedir function you talk about?   Well it's a feature in ONTAP that created shares and exports on the fly based on the client's usernames that are presented to ONTAP during the mount or drive map process.  This makes it so much easier to provision users, home directories, permissions, quotas, and everything in between. Take the guess work out of it all!

Does it help?
Yes!  There is actually a recent Microsoft Windows bug rolling around in build 1803 that can cause some performance issues.  When you use that pesky little search bar in Windows Explorer and search a mapped drive, and that mapped drive is actually a folder within an actual share, then Windows does some real deep searching all the way back to the root of the share.   WHAAA???  There are other stupid things happening with the Windows clients too that can cause some weird performance issues when there is a lot of file enumeration going on.  Most of this client behavior is happening on the home drive, or even worse, a roaming profile drive.  This is because when you create a home directory share, you create a single folder, share that folder, create everyone's home directory off that folder, then have the users map the folder level.  (something like mapping H: to  \\server\homedirfolder\username).  No I'm not going to get into the minutia of the behavior since MS has done some work to fix it, but it's there nonetheless. Let's all use this awesome ONTAP feature to mitigate these issues.

Do I really need to know how it works?
Maybe.   It's actually not that complicated.  When a user connects to the "homedir", ONTAP just looks into a few volumes specified for that user's name on a directory and then creates a hidden share on the fly and lets the user attach to it.  That's it.  It's that simple.  Nothing more, nothing less.

Why do I want it?
There's a few things that make this so much more simple than the traditional way of creating a "homedir" share and mapping sub-directories to it.

First of all, it's simpler to expand.  What if the volume serving your homedir is full?  What if it needs more performance?  What if we need to create more volumes?  Split it up?  You can see the problems there.  Let me give you a quick demo.

ONTAP System Manager is showing me that I only have 4 homedir volumes in the searchpath.  I want to add another.  Let's say we're bringing on more staff.

Let's just add the volume to the search path....

And voila!!!  It's there!









That was easy!   As long as all the homedir volumes are in the same SVM, then you can add as many volumes as you want to the search path.

Why else do I want this?  Well, the only thing you have to do when provisioning a new user is to create the user's homedir folder in one of the volumes of the search path.  No need to create a new share... no need to create a new volume.... no need to do anything else except create the folder and apply the permissions!  You can even automate this with PowerShell!  You can even balance the amount of homedirs in each volume with the PowerShell script!!!!

I'm Sold!  How do I implement this?
It's easy.  Just create the homedir share and add volumes to the search path.  With System manager... its an all-in-one process!!!

Create the home directory share.

Here's where you can get creative.

For the name, this is what you will be putting into AD for the home dir path.  There is a caveat though.   If you create a non-unique name and have more than 1 user log into a machine at the same time, then there can be issues.  Why?  MUP cache.  What???   Yes. I said MUP cache.  What in the blazes is that?  Well, when the client mounts the home directory (let's just call it \\svm\cifshomedir), what is going on under the covers is some DFS magic to actually point it in the right direction.  The clients cache this DFS magic.  Unfortunately, that caching happens at the machine level.  So when user jimbobbillyjo logs into the Windows machine and has their homedrive connected, then the client machine's DFS cache says that \\svm\cifshomedir is actually \\svm\jimbobbillyjo$.   Now when user georgemcf logs into the same machine (like with a user switch) then when it tries to map \\svm\cifshomedir, then it's actually trying to mount \\svm\jimbobbillyjo$.  That won't work for poor georgemcf.

In order to just bypass all that nonsense, use the appropriate abbreviation in the name.  %w will be the most common, then you can use %username% in the drive mapping in AD.

So what do I need for AD?

Just set up the homedir.  There are several schools of thought.  

* Home Folder:  This is the most obvious.  ADUC gives you a drive letter and path.  This will also give you some variables once logged in like %HOMEDRIVE%, %HOMEPATH% and %HOMESHARE% that you can use to do more with.

   For automation, you can use variables.  I would use \\svm-name\%username%.   Then ADUC will translate that to username.

* Roaming Profile:  Its the exact same as the Home Folder above, but its called the Profile path in ADUC. 

* GPO for mapped drives:  This is more of a fan favorite. No need to set something for every computer account.  If this is a domain-wide GPO, then everyone gets the drive mapped.  This is where you need to use the variables though.

  The downside?  You don't get any of the cute variables.

FlexCache now FREE!

Recently NetApp reintroduced FlexCache into ONTAP where both the origin and the cache can run on ONTAP 9.5 or higher.  That did come with a license; term & capacity based.  Well, the great folks at NetApp have decided that it should be part of the ONTAP package.  That means that its now freeeeeeeeeeee. 

Hold your horses.  That doesn't mean that you can now go creating FlexCache volumes all over the place in ONTAP.  There's no magic that can do that.  That just means you still need to get a license, but it costs $0.00!  Go bug your NetApp account team to go get you one.  I
n a future ONTAP version, you will be able to create FlexCache volumes without a license.  This may even be a P release!  (All the more reason to upgrade to P releases)

You want to kick the tires?  Go on over to https://www.netapp.com/us/info/what-is-flex-cache.aspx and see what you can do.

LDAPS is dead! Long live LDAPS!

So, as a TME for NFS, Name Services and SMB, I always get the question...

Does ONTAP support LDAPs??

StartTLS is insecure compared to LDAPS, right?

Well, let me tell you something........

LDAPS is sooo 90's.  There!  I said it!  90's?  really?   Yep!   RFC1777 was published in 1995!  And guess what?  There was no standard written for LDAP over SSL!  This was just assumed and taken from HTTPS at the time. The LDAP vendors took it, and rolled with it.

Enter LDAPv3.

RFC2251 was started and completed in 1997.  That didn't have any additional verbiage for on-the-wire encryption, but when RFC2830 was written in 2000, it made an extension for LDAPv3 specifically for StartTLS.  RFC4511 combined them into one LDAP v3 standard in 2006.  Once LDAP v3 was ratified and made it into mainstream in the early 2000's, LDAP vendors started to deprecate the LDAPS encryption method for StartTLS.

Deprecate.   That's a big word.   But what that means is that to "be usable but regarded as obsolete and best avoided, typically due to having been superseded."  Don't believe me?   Check it out:
https://www.openldap.org/faq/data/cache/605.html
https://directory.apache.org/api/user-guide/5.1-ldaps.html
https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol#Protocol_overview
I can go on....  Just do a Google search for LDAPS deprecated and see.

Clustered Data ONTAP started without any LDAPS support in favor of StartTLS because of the above reasons.  All LDAP vendors had to support the StartTLS extension because it was part of the standard.  Also, because LDAPS was deprecated, it could be yanked at any time.

But....  StartTLS is not as secure as LDAPS!!  It runs over a plaintext port!

Well, if you leave your LDAP install wide open, sure.  Your LDAP is still insecure.  But there are countermeasures that you can deploy to make sure StartTLS is secure.

1.  Make sure that StartTLS is enabled.  DUH.
   This requires certificates.   Certificates are hard, but learn them.  They make things secure.  For internal, self-signed certs are OK as long as the domains and permutations of them are accounted for.
2. Prevent queries without StartTLS.
   Now that StartTLS is enabled, you can set up LDAP servers to prevent LDAP commands unless the connection is StartTLS upgraded.  This can be done on a DIT (directory information tree) basis. 
3. Prevent LDAP binds without StartTLS
   There is also a way to prevent BINDs from happening in plaintext.  Consult your LDAP vendor's documentation on how to do this.
4. Prevent LDAP clients to send plaintext creds
   You still want to prevent your clients from sending their credentials in plaintext.  This means setting the minimum bind level at SASL.  The credentials passed will also be encrypted.

Doing all this will ensure that the LDAP communication is always secure.

How do you configure LDAP to work with StartTLS in ONTAP?  Easy.  Install the root certificate for the LDAP server's enterprise and then change the LDAP client config for -use-start-tls true and you are set to go.
Here's how your traffic looks: 

I don't see any holes in there.  Application data means its encrypted and is all buttoned up!  

What about ONTAP and LDAPS?   Well...  low and behold......   9.5 introduced the ability to do LDAPS.   Even though its deprecated and there are alternatives, the ability to do LDAPS was granted.
How do we do it?
Change the port in the ldap config to port 636.  Make sure -use-start-tls is turned to false and the same enterprise root cert is installed into ONTAP.  Is this more secure? Less secure?  Who knows.  Not my debate.
Here's what your traffic looks like:

You decide.  They look almost identical to me except for the LDAP START_TLS packets at the beginning.

FlexCache is BAAAAAAACK!!!!

We're starting to hear alot of "Global Namespace" in the industry.  Buzz, buzz, buzz.  Well, what really is a "Global Namespace"?   Well, it can be broken into 2 categories.

1. Your whole data structure is under one "root" volume and the data is actually stored all over the place.  But it's all under one directory tree with soft links, symlinks, junction points, widelinks, and a whole bevy of tricks.
   
   
   OR
2. Your dataset is available in multiple places in a read-write configuration (caching) and no matter where you are in the company's network (via VPN, overseas, in a remote office, at the datacenter) you get a similar response time to that dataset and its all read-writable.

Which one do YOU want?

The first one SOUNDS great, but what benefit does it have?   Well, now you don't have to go search the P: drive AND the M: drive AND the T: drive for what you want, you just have one MASSIVE-looking drive.  But your still searching through the subdirs to find your data.  It's nice... but just smoke and mirrors and your gaining VERY little.

On the Linux side, its a LITTLE better since your automounter just mounts stuff automagically under some path (maybe /data) and everything underneath looks like a single file system.  BUT you have the same Windows dilemma.  Is it REALLY better for your clients?  Also think of applications that now have to go down some unGAWDly path to get at its data.

THEN you have the whole performance issue.  Your Windows P: drive, which is the same as the Linux /data/pdrive path is hella ethernet miles away and performance SUUUX.  

Sooooooooo

Let's look at the second one.  Now this sounds good, but is it?  Well, this means that your data is in multiple places at once and that when you look at it from anywhere in the world, the performance is "local" and its a writable copy.   Awesome!  How does all this coordinate?  Well, ONTAP has a cool new thing called FlexCache.  Well, is it really new?   Well, yes it is. But it isn't.  

We had something in 7-mode called FlexCache.  It was good for its purpose, but lacked enhancements.    Bring the data in a read-heavy, but still writable capacity closer to whatever needs it. It was great for what it did, but the potential to move it forward and give it more features was limited by the backend technology.  Enter FlexCache in ONTAP starting with 9.5.  Its a whole new rewrite of the technology which can give so much more!

First of all, what are the use cases?  There are a few.  The first ones that come to mind is working on the same data in multiple places.  This is an example of AI, EDA, Media Rendering, Code distribution, and other similar workloads.  You have a single "master" data set, but you want other pieces to be able to read the same data and write some results or something into that same dataset.  The next one that comes to mind is creating multiple read-heavy copies to keep the "hot volume" syndrome down.  We all know there's challenges when multiple clients are reading the exact same file(s).  So let's spread the load!

So...  how do I get it and use it?
Starting in ONTAP 9.5, the new FlexCache is available. Give it a spin.  Let your mind explore the options you can use this new feature for.

I do have a Technical Report for the deep dive.  You can find it here:  https://www.netapp.com/us/media/tr-4743.pdf

Go for it!